AI Is Lowering the Barrier to Online Abuse. Platforms Need to Raise Their Defenses.

Bad actors used to need expertise and resources to scale online abuse. Now they just need access to basic AI tools.

In conversation with Martechify, Cinder co-founder and CEO Glen Wise joined Alexandra Ryabova, Head of Operations at Wizz App, to discuss how AI-powered impersonation, synthetic identities, phishing, and other forms of abuse are reshaping the trust and safety landscape.

The important shift isn’t necessarily the emergence of entirely new threats. It’s how accessible existing tactics have become.

The barrier to entry for conducting highly sophisticated attacks has dropped down to basically zero.

Generative AI can give bad actors capabilities that once required considerably more expertise and resources. Creating a convincing fake identity, writing a polished phishing message, or producing synthetic content now takes minutes. The challenge for Trust and Safety teams is keeping pace with that acceleration.

It’s a problem Wise has been thinking about since before Cinder existed. While working on Meta’s threat intelligence team, he saw firsthand how difficult it could be for large organizations to respond quickly to emerging threats. Identifying a new form of abuse was only the beginning. Teams then had to develop policies and turn those policies into effective enforcement.

There must be a much faster way of doing it than taking nine months to go through that full cycle.

That experience became part of Cinder's mission: To help trust and safety teams quickly move from identifying a threat to actually doing something about it.

But moving faster doesn’t mean abandoning thoughtful decision-making. Wise emphasized that effective enforcement starts by deciding what a platform actually wants to allow: “Policies really are the center of the universe.”

Clear policies give platforms a foundation for determining what should be automated, what requires human judgment, and how the quality of those decisions should be measured. Instead of starting with a model or moderation tool and working backward, teams can start with the experience they want to create and build enforcement around it.

AI-generated content also makes context increasingly important. A convincing profile image or message may look legitimate in isolation. Detecting abuse requires platforms to consider behavioral signals alongside the content itself and to think about how an adversary might attempt to exploit their systems. That combination of policy, technology, behavioral signals, and human judgment is becoming increasingly important as platforms contend with abuse at greater scale.

While AI has changed the economics of online abuse, the fundamental responsibility of platforms hasn’t changed. Safer online experiences require companies to make deliberate choices about what they will tolerate, build systems capable of enforcing those choices, and continually adapt as adversaries change their tactics.

The technology will keep evolving. Trust and Safety teams need the infrastructure to evolve with it.