The Best AI Starts with Human Judgment

My job didn’t exist just one year ago.

I'm an Agent Product Manager at Cinder, and I design how our AI agents fight internet abuse. I turn enforcement policy into agent architecture, define agents' context and capabilities, and establish the boundaries of what they can do without a person. Then I dig into what they miss and why and redesign from there.

I spoke on a panel titled “What’s Happening To and At Work?” at the All Tech Is Human and Notre-Dame Tech Ethics Lab’s Agentic AI workshop. At the panel, I shared some of what I’ve learned in my new role at Cinder.

Agents work best when they work with humans

People are skeptical of AI’s judgment, and risk tolerance is low when stakes are high. It’s one thing to let an agent build your go-to-market strategy, but it’s another entirely to trust an agent to decide whether a piece of content is designed to manipulate elections, or whether a user is expressing an intent to harm themselves or others.

Though more AI-native teams trust AI to make the right decisions, legacy teams with long-established processes are more likely to express skepticism and resistance. They often assume automation is designed to entirely replace human judgment.

I work hard to correct that assumption when working with our clients. Agents work with humans, not just for them. We find the strongest systems use agents to handle obvious cases while surfacing tough decisions to human reviewers. In these systems, AI agents review cases involving extremely graphic content, shielding human moderators from the psychological toll of repeated exposure to the worst abuse on the internet. Instead, the agent flags borderline cases and empowers human reviewers to do what they do best: Use their judgment and expertise to make difficult decisions.

We must build safeguards into agents from the start

When human collaborators work cross-functionally to build something, there are layers of doubt, attention to detail, and competing priorities that typically catch missteps before they get too far. Most teams’ output relies on more than just a single individual’s assurance.

It’s tempting to assume agents work like humans, but in reality, they often rely on hard-coded assumptions that confidence equals correctness.

Without foresight, an agent’s confident but incorrect decision can rush through the pipeline where a human teammate would have intervened and asked to double-check. Safeguards need to be explicit and planned ahead, whether that means routing decisions based on stakes, escalating disagreements, running proactive evals, or putting agents in shadow mode until we’re entirely confident that they can perform up to our standards.

Where does the agent’s judgment end and the human’s begin?

I closed the panel by asking a question I didn’t think anyone in the room could answer: Where does the agent’s judgment end and the human’s begin? The goalposts change day to day. Agents get better by learning from human decisions, and humans see how their guidelines are driving agents’ behavior, which gives them more insight into their own blind spots. This ongoing conversation between agents and humans has the potential to accelerate our understanding of how our own confidence and assumptions drive decision making.