How to Set Up an AI Red Teaming Program for a Generative AI Product

AI red teaming is structured adversarial testing of a generative model. Experts attack it the way real abusers will, so you find the failures before the adversary does. If you are new to the term, read What Is AI Red Teaming? first. This guide covers how to build the program.

Why do companies red team generative AI models before release?

Three reasons come up most often.

• Catch failures while they are cheap to fix. A safety gap found in development can be addressed in training, filtering, or a targeted safeguard. The same gap found after launch becomes an incident.
• Give customers evidence. Enterprise buyers increasingly ask for external validation of a model's safety claims. Krea's enterprise customers wanted this for Krea 2's internal safety benchmarks, and Cinder's engagement addressed that demand.
• Take responsibility for deployment. This applies strongly to open-weight models. Their flexibility moves more of the safety responsibility to whoever deploys them, so teams running models themselves need to know where the safeguards fail.

Step 1: Define the harms in scope

Start with the harms your model could facilitate in your product and market. Write them down as a list, with a short definition of each. Common categories include:

• Child safety and CSAM
• Non-consensual intimate imagery (NCII)
• Violence facilitation and terrorism
• Fraud and scams
• Manipulation and deception

Keep the definitions tied to your policies. A test is only useful if everyone agrees on what counts as a failure. Sensitive categories need reviewers who have handled this material professionally, with proper safeguards for the people doing the work.

Step 2: Decide what to test and under what conditions

Make a list of the model versions and setups to test, and write down the conditions so the test can be repeated.

• Checkpoints. Test more than the final release. In the Krea 2 engagement, Cinder evaluated both raw and post-trained checkpoints. Comparing them shows what training and safety work changed.
• Open-weight variants. If you release weights, test the original model and modified versions, since modification can remove safeguards.
• Prompting conditions. Decide whether a system prompt is present. In Cinder's open-weight study of six models from six developers, ranging from 1B to 34B parameters, each model received every prompt once and no system prompt. Holding conditions constant makes results comparable across models.
• Capabilities. Include tools, agents, image generation, or other features that expand what the model can do.

Step 3: Build realistic threat scenarios

Test the way a motivated abuser would act. A list of well-known jailbreak phrases will find well-known failures. Realistic scenarios cover the whole path to harm: what the person wants to accomplish, what information or output they need at each stage, and how they would rephrase after a refusal.

For violence facilitation, Cinder's evaluation scores harmful uplift as a yes-or-no result at each stage of a plan, rather than measuring severity. Both a vague materials list and a step-by-step procedure can count as uplift. A simple binary rule keeps reviewers consistent and avoids arguments over how bad is bad enough.

Step 4: Use expert reviewers and classify every result

Automation can generate and run thousands of prompts. Judging the outputs often needs human experts, especially in context-heavy categories. A strong review process does three things:

• Reviews every generation. For Krea 2, expert human labelers reviewed every generation.
• Classifies by harm category. Labels by category let you see where the model is weakest.
• Separates unprompted from prompt-requested violations. A model that produces harmful content when nobody asked for it has a different problem from a model that complies with a harmful request. Each calls for a different fix.

The output is a labeled dataset. It becomes training and evaluation material for the next round.

Step 5: Turn findings into safeguards

Findings only matter if they change the model or the product. Work through them in this order:

• Look for patterns. Group failures into vulnerability patterns. In the FLUX.2 engagement, Cinder categorized unsafe outputs, analyzed successful defenses, and shared vulnerability patterns with Black Forest Labs' research team during development.
• Prefer targeted fixes. Broad refusals can make a model less useful. A targeted safeguard addresses a specific risk and can be tested.
• Write the policy insight down. Failures often show where policy is unclear. Updating the policy helps reviewers and the model stay aligned.

Step 6: Re-test and measure

Findings that don't change the model are just paperwork. After a fix, run the same tests again and compare the numbers, so you ship with evidence. Short cycles help: the Black Forest Labs engagement included rapid red-team cycles with turnaround times under 48 hours.

The measurable result is the point. By the end of its FLUX.2 engagement with Cinder, Black Forest Labs reported a greater than 90% reduction in CSAM and NCII vulnerability, and FLUX.2 launched 10x safer than benchmark industry models.

How often should you run it?

• Before every major release
• After a meaningful change, such as a new model version, fine-tune, new capability, or new market
• On a recurring schedule between releases, using re-runnable evaluations to catch regressions

Build in-house or buy?

Many teams combine both. An in-house group helps define needs and priorities, and an external team runs deep, independent engagements before launches. For help choosing a partner, see our guide on how to choose an AI red teaming vendor.

A starter checklist

• List harm categories and write a definition for each.
• List model versions, checkpoints, and variants to test.
• Fix the test conditions and write them down.
• Draft threat scenarios for each harm category.
• Choose reviewers and define the labeling scheme.
• Decide how results map to fixes and who owns them.
• Schedule the re-test and set the metrics you will track.
• Decide what evidence you will share with customers and leadership.