How to A/B test AI prompts and compare outputs
One good answer does not establish that a prompt will work on the next input. Test a small set of representative cases and write down what a good result looks like before changing the prompt.
- Choose a typical input, a short one, and an ambiguous one. Set checks for factual accuracy, adherence to your format, and how the answer handles missing facts.
- Keep the model and input constant when comparing prompt versions. When comparing models, keep the prompt and input fixed so you can see what the model change did.
- Use the Pro testing and comparison tools to review outputs side by side. Keep the version that performs well across your cases and inspect any result that still needs human review.
A real A/B test for customer support summaries
We ran both prompts on the same three fictional support tickets using the Gemini model configured for Prompt Builder. To reproduce the test, paste each prompt into a separate run, add a blank line and the same ticket, and keep the model and settings fixed. The prompts, tickets, and outputs below are in their original English.
gemini-3.1-flash-lite-previewTemperature 1, high thinking, 1,024-token output cap
Prompt A
Summarize this customer support ticket in exactly two plain-text lines. Use the labels Issue and Order number.Prompt B
Summarize this customer support ticket in exactly two plain-text lines. Use the labels Issue and Order number. Write Unknown for the order number when the customer did not provide one.Award one point per ticket if the second line contains the exact expected value after “Order number: ”, including punctuation. Use the same rule for both versions.
| Case | Expected value | A | B |
|---|---|---|---|
| Order number given | PB-2481 | Pass | Fail |
| Order number missing | Unknown | Fail | Pass |
| Support ticket number only | Unknown | Fail | Pass |
| Total matches | 1/3 | 2/3 | |
B returned the required Unknown value on both tickets with no order number. It added a period to the supplied number, which failed the exact-match check. Each version ran once per ticket. Repeat the test on your own inputs before adopting a prompt.
Order number given
Ticket input
My blue travel mug arrived cracked. My order number is PB-2481.Actual output from A
Issue: Blue travel mug arrived cracked
Order number: PB-2481Actual output from B
Issue: Customer received a cracked blue travel mug.
Order number: PB-2481.Order number missing
Ticket input
My blue travel mug arrived cracked. Can you send a replacement?Actual output from A
Issue: Customer received a cracked blue travel mug and requested a replacement.
Order number: Not provided.Actual output from B
Issue: The customer received a cracked blue travel mug and is requesting a replacement.
Order number: UnknownSupport ticket number only
Ticket input
My blue travel mug arrived cracked. Please see support ticket 5176 for the photo. Can you send a replacement?Actual output from A
Issue: Customer received a cracked blue travel mug and is requesting a replacement.
Order number: Not provided (referenced support ticket 5176).Actual output from B
Issue: Customer received a damaged blue travel mug and is requesting a replacement.
Order number: UnknownCan I test prompts without using the built-in runner?
Yes. Copy each version from the preview to your model of choice and apply the same criteria manually. In-workspace AI testing and comparison are Pro features.