Generative media is becoming infrastructure. Safety must keep pace.

The acceleration in generative AI adoption is coming from an unlikely source, according to a new report out from Cinder customer fal.

In State of Generative Media, Volume 2, fal details rapid expansion in the adoption of generative models. The growth, the company said, is coming from enterprises that previously had little connection to generation. Entertainment, gaming, and advertising companies are now developing sophisticated workflows that look more like something you'd find at a specialist lab. They are chaining models together, adapting them for specific tasks, and embedding generation directly into production. In just the last six months, active API users on fal have doubled.

fal argues that generation is becoming infrastructure. Put another way: Every company is becoming an AI company.

This has critical implications for Trust and Safety work. The tension between open-weight models, zero data retention, and safety is already driving conversation. Companies that choose to ship on open-weight models must then develop their own safety infrastructure. As more companies utilize or post-train open-weight models, stronger, evolving Trust and Safety parameters need to be considered alongside the progress being made.

Companies need developed solutions built with leading partners across the AI ecosystem to address this on a holistic scale. Those solutions will need to be multifaceted and nuanced enough to fit each company's needs, including partners like Cinder that can enable Trust and Safety at the speed the industry is shifting.

Faster generation changes the safety workload

The report highlights the ever-increasing speed at which generative models operate. What used to take minutes now takes seconds, and while that speed lets companies use AI in new ways, it also accelerates threats.

"When the tools to create anything become available to everyone, and the output moves faster than any process designed to review it, the harder questions about trust tend to get set aside for later," fal's Head of Trust and Safety Sean Bonawitz wrote in the report. "What we are seeing now, across the industry and in our own work, is that later has arrived."

Bad actors can generate material faster than ever before, using tools that involve multiple models, transformations, and types of media. Safety operations have to account for that change. Reviewing individual outputs is only part of the work, and teams also need to understand how the system as a whole can be misused.

The report points to a concerning industry signal from the Tech Coalition's Lantern program. In 2024, participating companies shared 327 signals tied to AI-generated or AI-manipulated child sexual abuse material. In 2025, that number rose to 5,641, a 17-fold increase in one year.

These figures underscore the need for coordinated detection and response at every company using generative models.

Connecting detection to action

fal describes a safety approach that brings together several capabilities, including automated CSAM hash-matching through NCMEC and onboarding with StopNCII.org. Cinder provides the structured environment that connects detection, review, and enforcement.

In the report, Bonawitz wrote, "Cinder ties it together: one structured place to run detection, review, and enforcement, so that knowing about a violation and acting on it are not separated by the gaps that open when a process lives across five tools and someone's memory."

That structure matters because identifying a potential violation starts a process. A team still needs to assess the evidence, apply its policies, and take the appropriate action. When those steps are scattered across tools, context can be lost and follow-through becomes harder.

Bringing the work together helps teams turn individual detection signals into a consistent operational response. As generation expands across models and product surfaces, that coordination becomes an essential part of any generative product.

The report also highlights Cinder's role in another part of the safety lifecycle: evaluating models and testing whether mitigations work.

Ahead of its FLUX.2 release, Black Forest Labs partnered with Cinder on third-party red-teaming involving nearly 4,000 adversarial attacks across text-to-image and image-to-image capabilities. The work tested checkpoints before, during, and after mitigation.

Black Forest Labs reported that targeted post-training reduced vulnerabilities by 77-98% compared with earlier checkpoints.

This is the value of repeatable evaluation: teams can identify weaknesses, assess changes, and make decisions with better evidence.

"That there is now a shared technical vocabulary for this work, that a company can red-team against a common framework and publish the results for others to scrutinize, is itself a sign of the reckoning maturing," Bonawitz wrote.

Safety is an ongoing operational practice

fal's report describes an industry moving toward specialized models, connected workflows, and generation embedded in everyday products. Safety must evolve alongside those systems, with clear processes for testing, reviewing, and acting.

Model evaluations help teams understand vulnerabilities before release, and production operations help them detect and respond to abuse in use. Each provides evidence that can inform future improvements.

At Cinder, we're proud to support fal in that work. Their report shows what progress looks like in practice: measurable testing, connected operations, and candor about the challenges still ahead.

Read fal's State of Generative Media, Volume 2.