Skip to main content

How to A/B test AI prompts and compare outputs

One good answer does not establish that a prompt will work on the next input. Test a small set of representative cases and write down what a good result looks like before changing the prompt.

  1. Choose a typical input, a short one, and an ambiguous one. Set checks for factual accuracy, adherence to your format, and how the answer handles missing facts.
  2. Keep the model and input constant when comparing prompt versions. When comparing models, keep the prompt and input fixed so you can see what the model change did.
  3. Use the Pro testing and comparison tools to review outputs side by side. Keep the version that performs well across your cases and inspect any result that still needs human review.

A real A/B test for customer support summaries

We ran both prompts on the same three fictional support tickets using the Gemini model configured for Prompt Builder. To reproduce the test, paste each prompt into a separate run, add a blank line and the same ticket, and keep the model and settings fixed. The prompts, tickets, and outputs below are in their original English.

gemini-3.1-flash-lite-previewTemperature 1, high thinking, 1,024-token output cap

Prompt A

Summarize this customer support ticket in exactly two plain-text lines. Use the labels Issue and Order number.

Prompt B

Summarize this customer support ticket in exactly two plain-text lines. Use the labels Issue and Order number. Write Unknown for the order number when the customer did not provide one.

Award one point per ticket if the second line contains the exact expected value after “Order number: ”, including punctuation. Use the same rule for both versions.

Exact matches for the order number field
CaseExpected valueAB
Order number givenPB-2481PassFail
Order number missingUnknownFailPass
Support ticket number onlyUnknownFailPass
Total matches1/32/3

B returned the required Unknown value on both tickets with no order number. It added a period to the supplied number, which failed the exact-match check. Each version ran once per ticket. Repeat the test on your own inputs before adopting a prompt.

Order number given

Ticket input

My blue travel mug arrived cracked. My order number is PB-2481.

Actual output from A

Issue: Blue travel mug arrived cracked
Order number: PB-2481

Actual output from B

Issue: Customer received a cracked blue travel mug.
Order number: PB-2481.
Order number missing

Ticket input

My blue travel mug arrived cracked. Can you send a replacement?

Actual output from A

Issue: Customer received a cracked blue travel mug and requested a replacement.
Order number: Not provided.

Actual output from B

Issue: The customer received a cracked blue travel mug and is requesting a replacement.
Order number: Unknown
Support ticket number only

Ticket input

My blue travel mug arrived cracked. Please see support ticket 5176 for the photo. Can you send a replacement?

Actual output from A

Issue: Customer received a cracked blue travel mug and is requesting a replacement.
Order number: Not provided (referenced support ticket 5176).

Actual output from B

Issue: Customer received a damaged blue travel mug and is requesting a replacement.
Order number: Unknown

Can I test prompts without using the built-in runner?

Yes. Copy each version from the preview to your model of choice and apply the same criteria manually. In-workspace AI testing and comparison are Pro features.