Safety by Design: One year of progress protecting children in the age of AI

Generative AI is transferring quick. It’s reshaping how we work, join and live- and it carries actual dangers for kids. At Thorn, we imagine it is a pivotal second: security have to be in-built now. 

That’s why, in April 2024 we partnered with All Tech is Human to launch our Security by Design for Generative AI initiative. Collectively, we introduced among the most influential AI corporations to the desk, securing commitments to forestall the creation and unfold of AI-generated baby sexual abuse materials (AIG- CSAM) and different sexual harms in opposition to youngsters. 

Right now, we’re proud to share the primary annual progress replace. This replace highlights actual wins achieved by this initiative*, progress from trade leaders, new requirements shaping the sphere, and the worldwide coverage milestones which might be starting to shift expectations.

That is value celebrating. For the primary time, main voices within the AI ecosystem have dedicated not solely to tackling the danger of AI-generated baby sexual abuse materials (AIG-CSAM) and different sexual harms in opposition to youngsters, but in addition to documenting and sharing their progress publicly.

What we’re studying collectively

Simply as necessary because the progress itself are the collective learnings from this primary yr. Throughout corporations and the AI ecosystem, we’ve seen the place safeguards work, the place they fall brief, and the place new dangers are rising. These insights are serving to to form a clearer image of what it’ll take to maintain youngsters secure within the age of generative AI.

The next sections have fun the progress we’ve seen, and in addition level to the teachings that can information the subsequent stage of this work.

Progress value celebrating: What’s modified this yr

Firms detected and blocked lots of of 1000’s of makes an attempt to generate dangerous content material.Lots of of fashions that have been constructed to create baby abuse imagery have been faraway from platform entry.A number of coaching datasets at main AI corporations have been filtered for CSAM and different abusive materials.Tens of millions of prevention and deterrence messages surfaced in response to violative prompts.Lots of of 1000’s of studies of suspected baby sexual abuse materials—masking AI-generated and non-AI content material—have been filed to NCMEC’s CyberTipline.Firms invested vital assets in new protecting applied sciences and analysis.

These are actual wins. They show that, when applied with intention and rigor, Security by Design is efficient. However we’re additionally seeing clear patterns in what’s lacking.

What we’re studying

NSFW ≠ CSAM. Generic grownup content material filters used to wash coaching information might miss vital alerts particular to CSAM and CSEM, together with low-production makes an attempt to obscure origin. CSAM-specific tooling and knowledgeable evaluation are vital.Eradicating CSAM from coaching datasets isn’t sufficient. Compositional generalization continues to be an actual threat. If fashions be taught dangerous associations, like benign pictures of kids + grownup sexual content material, they might produce abusive outputs even with out express coaching information. Extra corporations want mitigation methods right here—throughout all modalities, not simply audiovisual.Unbiased analysis issues. Third-party assessments broaden protection and floor blind spots that inside groups might miss. That is very true given authorized constraints on dealing with CSAM. We want structured public-private partnerships to make sure testing could be carried out lawfully and safely.Provenance have to be adversary-aware. Some approaches (like easy metadata) have worth however are too straightforward to strip. Adoption throughout open-source releases is sort of zero. With out sturdy, interoperable, and confirmed provenance for brazenly shared fashions, we’ll proceed to play catch-up.Open weight ≠ open season. When corporations launch open-weight fashions, they have to carry ahead the identical safeguards they apply to closed methods—e.g. coaching information cleansing, person reporting, and documented misuse testing. Additionally they have to put money into researching options which might be sturdy to the distinctive dangers of open weight (e.g. adversarial manipulation downstream). Internet hosting and search are untapped chokepoints to forestall hurt. Third-party mannequin hubs and engines like google will not be persistently pre-evaluating uploads or de-indexing abusive instruments (e.g., nudifiers, or chatbots with grownup personas that sexually have interaction with baby customers). That makes dangerous capabilities too straightforward to seek out and use.Agentic and code-gen methods want a plan. Agentic AI permits LLM’s to regulate workflows and work together with exterior environments and methods; code-generation permits builders to generate code by prompting LLM’s. We’re not but seeing significant mitigations to forestall misuse of those instruments to e.g. construct nudifiers or to automate sextortion schemes.Reporting wants extra granularity. Hotlines and firms ought to clearly flag (after they have this info) whether or not content material is AI-generated, AI-manipulated, or unaltered (in addition to info like prompts and mannequin info). This element helps legislation enforcement triage and prioritize real-world sufferer identification.

Setting the usual for security

Altering how particular person corporations act is vital, however it’s not sufficient. To actually defend youngsters, we want shared guidelines of the highway that increase the baseline for the entire trade. That’s why standard-setting and coverage engagement are such an enormous deal: they make baby security a requirement, not only a alternative.

This yr, we have been proud to see Thorn’s perspective included in main requirements and coverage milestones all over the world—from international technical our bodies to landmark regulatory steering. These inclusions replicate actual steps towards making baby security a basis of how AI is constructed and ruled.

Requirements

IEEE Really useful Observe

IEEE develops most of the requirements individuals depend on day by day, like Wi-Fi and Bluetooth. 

In 2024, Thorn led the trouble to ascertain an IEEE working group, to draft the first worldwide normal embedding baby safety throughout the AI lifecycle. This Really useful Observe will formalize Security by Design as a worldwide finest apply, drawing on enter from technologists worldwide. 

In September 2025, the IEEE working group advancing this normal voted to ship the draft Really useful Observe to poll.  Embedding Security by Design as an IEEE normal will present a stable basis for safer improvement, deployment, and upkeep of generative AI fashions.

NIST AI 100-4

The Nationwide Institute of Requirements and Expertise (NIST) is a part of the U.S. Division of Commerce. Its mission is to advertise U.S. innovation and industrial competitiveness by advancing measurement science, requirements, and expertise in ways in which improve financial safety and enhance high quality of life. NIST’s requirements are extremely revered and globally influential. 

NIST’s AI 100-4 normal is concentrated on decreasing dangers from artificial content material, together with AIG-CSAM. Thorn was invited to supply early suggestions and deep steering into this groundbreaking AI normal that helped form new U.S. finest apply on decreasing dangers from artificial content material.

Coverage engagement

EU AI Act (2025): Thorn was one of many civil society organizations contributing to the EU AI Act Code of Observe, EU’s first main regulatory device for AI. Our enter helped safe key wins—like requiring corporations to doc how they take away CSAM from coaching information and deal with CSAM as a systemic threat.World briefings: We’ve hosted webinars for policymakers within the U.S., U.Ok., and EU, and our work was acknowledged by the White Home as a part of efforts to fight image-based sexual abuse.

These advances don’t clear up the issue on their very own, however they set clear expectations and create accountability, guaranteeing baby security is constructed into the muse of AI.

What the ecosystem should do subsequent

Defending youngsters on this digital age is everybody’s job. Expertise corporations have a vital position, however they will’t do it alone. Listed below are some steps we advocate for these within the broader baby safety ecosystem:

Builders: Use specialised CSAM detection instruments to wash coaching information, and incorporate safeguards to each closed and open fashionsPlatforms & search: Take away nudifiers, tutorials and abusive fashions from outcomes, consider third-party fashions and expertise earlier than offering entry, and make it straightforward to report and implement in opposition to dangerous content material.Coverage & governance: Give legislation enforcement the assets they want, replace legal guidelines to maintain tempo with AI, and permit secure partnerships for red-teaming.Faculties, caregivers & youth: Discuss brazenly about dangers, particularly as misuse shifts towards friends. Easy, judgment-free conversations go a good distance.Academia & trade: Spend money on analysis for extra sturdy provenance options and safeguards within the open-source setting, stronger detection of artificial content material, higher safeguarding of kids’s imagery, and safeguards for brand spanking new AI methods like brokers and code instruments.

What’s subsequent from Thorn

For greater than a decade, Thorn has helped the ecosystem get forward of rising threats to youngsters. We:

Conduct cutting-edge analysis to know the methods expertise is altering the panorama for on-line baby security.Companion with builders to embed Security by Design throughout the AI lifecycle.Present expertise (like Safer) and knowledgeable consulting to detect, evaluation, and report baby sexual abuse materials and different exploitative content material at scale.Convene cross-sector collaborations, contribute to and promote requirements, and help coverage that raises the baseline for everybody.

The progress we’ve seen this yr proves what’s doable when security is in-built from the beginning – and the gaps present how a lot work stays.

In the event you’re constructing or deploying AI, now’s the time to behave. Collectively, we will proceed to make sure much more progress—and fewer alternatives for hurt.

*DISCLOSURE: The data and conclusions contained on this weblog rely partly upon information that has been self-reported to Thorn by the businesses taking part within the Security by Design effort. Thorn didn’t independently confirm this information. For extra info relating to information assortment practices and use rights, please see the total report.