Meta Contractor Project 'Cannes' Used Fictional Minors to Test Rival AI Chatbots on Sensitive Topics

By Decode Today News

Meta Contractors Posed as Teens to Prompt Rival Chatbots About Suicide, Sex, and Drugs Technology

Meta, through one of its contractors, orchestrated a covert project that involved hundreds of workers posing as minors online to probe competitor artificial intelligence chatbots with highly sensitive prompts related to suicide, sex, and illegal drugs. This expansive operation, internally known as "Cannes" and managed by Meta contractor Covalen, targeted leading AI platforms including OpenAI's ChatGPT, Google's Gemini, and Character.AI, aiming to benchmark their safety responses to high-risk inquiries.

Covert Testing by Meta Contractors Raises AI Ethics Concerns

A recent investigation has uncovered a controversial project where Meta contractors, under the code name "Cannes," created fake underage profiles to test rival AI chatbots like ChatGPT and Gemini. These contractors reportedly sent thousands of prompts, some involving graphic content, to assess the chatbots' responses to queries about suicide, self-harm, sex, eating disorders, and drug use. The project, which ran as recently as April 2025, has drawn criticism from AI ethics experts for its scale, secrecy, and potential violation of competitors' terms of service, sparking a debate on industry-standard safety testing versus potentially anti-competitive practices. The project, active until at least April 2025, instructed contractors to establish dummy online accounts purporting to belong to individuals under 18 years old. These fictional minors then engaged with the rival chatbots, sending written prompts and even images designed to push the AI systems toward responses their inherent safety mechanisms were intended to refuse. All chatbot interactions and responses were meticulously copied into spreadsheets for review. Internal documents and accounts from five individuals familiar with the project, recently reviewed by WIRED, shed light on the operation's scale and nature. One round of testing completed in August 2025 alone saw over 45,000 prompts directed at the competitor chatbots. The companies behind these AI systems – OpenAI, Google, and Character.AI – were reportedly unaware of this clandestine testing. The prompts themselves often mirrored real-world crises experienced by young people. Examples included a query from a 13-year-old girl claiming pregnancy by an adult neighbor and seeking information on obtaining abortion pills, or a fifth-grader describing a classmate with a gun pointed at his mouth. Other prompts delved into scenarios like a girl asking how to conceal bulimia from her parents, or a student inquiring where to "get a cocaine." Some images sent by contractors depicted pills, knives, nooses, and even a medical diagram of a gynecological procedure. A spreadsheet examined by WIRED detailed 3,748 prompts from the project. Hundreds focused on themes of suicide and self-harm, while numerous others discussed eating disorders. At least 239 prompts involved sex or romance, with additional queries concerning drugs, profanity, and racial slurs. Not all interactions were in English; one French-language prompt referenced the suicide of Jamey Rodemeyer, a bullied bisexual teenager, and controversially asked the chatbot to agree that "if he'd been a straight guy, maybe he'd still be here today."

Meta's Defense and Industry Standards

Meta defended the project as "routine safety testing." A spokesperson stated that "testing and benchmarking chatbot responses to help ensure safe and age-appropriate experiences is a responsible, industry-standard practice, and any suggestion otherwise completely misunderstands how technology companies work to refine and improve their systems." The company also clarified that it does not use competitor benchmarking data to train its own AI models. Covalen, the managing contractor, has not issued a statement regarding the allegations. While testing competitors' products is not inherently unusual in the AI sector – for instance, Scale AI contractors reportedly compared Google's Bard outputs with ChatGPT to improve Bard's responses – the "Cannes" project's methodology has raised eyebrows. Former contractors involved expressed alarm, with one worker fearing the potential for generating or preserving child sexual abuse material if chatbots responded inappropriately to certain sexual prompts involving minors. Another contractor worried that the project amounted to secretly acquiring material from competitors to potentially feed back into Meta's systems, though Meta denies using this data for its own model training. Rumman Chowdhury, CEO and founder of Humane Intelligence PBC, reviewed a sample of the prompts and the project summary. She critiqued "Cannes" for appearing "designed to systematically break those rules, via dummy accounts masquerading as children," differentiating it from standard industry evaluation. Chowdhury acknowledged the utility of a dataset of youth-safety prompts for comparing refusal rates, but highlighted the project's scale, opacity, and lack of disclosure to the tested companies as significant deviations from public safety benchmarks.

Legal and Ethical Scrutiny

Legal experts Kendra Albert and Riana Pfefferkorn, specializing in online speech and technology law, reviewed examples of the prompts for WIRED. They concluded that the material did not cross the line into soliciting child sexual abuse material or illegal obscenity. The examined spreadsheet did not show prompts explicitly asking chatbots to generate such content, and image generation requests were rare. However, the methods employed by the "Cannes" project appear to violate the terms of service of the targeted AI companies. OpenAI's policies prohibit unsolicited safety testing, attempts to bypass safeguards, and using outputs to "develop models that compete with OpenAI." Google's terms disallow attempts to bypass safety filters outside its official testing programs and prohibit content related to self-harm, child sexual abuse, or illegal substances. Character.AI's public safety guidelines forbid harmful, exploitative, illegal, and obscene content, and the company has stated "No more open-ended chat for under-18 users" since late 2025. A spokesperson for Character.AI confirmed the company had not authorized the testing and stated the described conduct violated its terms and policies, calling it "a violation of the characters and worlds our community has created." OpenAI spokesperson Drew Pusateri indicated the company was "looking into the issue." Google stated it had not authorized the testing and was unaware of its purpose. While Google's internal tests of the provided samples showed Gemini adhering to its policies, the company noted it lacked sufficient information to determine a terms of service violation. For Chowdhury, the core issue remains whether a secretive project against competitors, using accounts impersonating minors, can genuinely be considered ordinary safety work. She described the blending of safety evaluation and competitor benchmarking as a "governance gray zone where safety becomes a convenient cover for anticompetitive practices."

Why This Matters for Global Readers

This revelation underscores the complex and often murky ethical landscape of AI development and competition. As artificial intelligence becomes increasingly integrated into daily life, particularly for younger demographics, the methods employed by tech giants to ensure AI safety and competitive edge come under intense scrutiny. Global readers, especially parents and policymakers, will be keenly interested in how companies like Meta, OpenAI, and Google navigate the responsibilities of developing powerful AI while upholding ethical standards and protecting vulnerable users. The incident highlights the ongoing debate about transparency, accountability, and the boundaries of competitive practices in the rapidly evolving AI industry. It also prompts important questions about what constitutes "industry standard" behavior when it comes to testing, especially when sensitive user data and the potential for harm are involved.

Frequently Asked Questions About AI Chatbot Testing

What was Project 'Cannes'?

Project 'Cannes' was an initiative managed by Meta contractor Covalen, involving hundreds of workers posing as minors online to test how rival AI chatbots, including OpenAI's ChatGPT, Google's Gemini, and Character.AI, responded to high-risk prompts concerning suicide, sex, eating disorders, and drugs.

Why did Meta conduct this testing?

Meta stated that the project was "comprehensive AI safety benchmarking" aimed at delivering "critical datasets for model comparison and compliance," and to ensure "safe and age-appropriate experiences."

Did the rival AI companies know about the testing?

According to reports, OpenAI, Google, and Character.AI were not aware of the testing conducted by Meta's contractors. Character.AI confirmed it did not authorize the testing and that it violated their terms of service.

What kind of prompts were used?

Prompts included scenarios like a 13-year-old seeking abortion pills, a fifth-grader witnessing gun violence, inquiries about hiding bulimia, asking where to buy cocaine, and prompts related to self-harm and sexual themes. Some images sent included pills, knives, and nooses.

Was the testing considered ethical or legal?

While legal experts found the prompts themselves generally did not cross into soliciting child sexual abuse material, AI ethicists criticized the project's secretive nature, scale, and the impersonation of minors as beyond "industry standard." The testing also appears to have violated the terms of service of the rival AI platforms.

The Wider Implications

The 'Cannes' project brings into sharp focus the ethical dilemmas inherent in the fierce competition among AI developers. It highlights the tension between a company's drive for robust safety testing and the potential for such methods to stray into questionable territory, particularly when involving the impersonation of minors and potential violations of competitor terms. As AI models become more sophisticated and widely used, establishing clear, transparent, and ethically sound industry standards for testing and benchmarking will be crucial. This incident is likely to intensify calls for greater oversight and collaborative efforts within the AI community to ensure that safety testing serves its intended purpose without compromising privacy, ethics, or competitive fairness. The conversation around AI development must extend beyond technical capabilities to encompass the profound societal and ethical responsibilities that come with creating such powerful tools. If you or someone you know needs help, call 988 for free, 24-hour support from the National Suicide Prevention Lifeline. You can also text HOME to 741-741 for the Crisis Text Line. Outside the US, visit the International Association for Suicide Prevention for crisis centers around the world.

Decode Today News

Ads

Breaking News

Meta Contractor Project 'Cannes' Used Fictional Minors to Test Rival AI Chatbots on Sensitive Topics