# LLM-as-a-Judge

LLM-as-a-judge uses a language model to score or compare outputs against an explicit evaluation rubric.

- Type: method
- Domain: Data, experimentation & growth
- Level: Advanced
- Last reviewed: 2026-10-06
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/llm-as-a-judge/>)

## What it means

A model judge receives a task, candidate output or pair of outputs, evaluation criteria, and any necessary reference evidence. It returns a rating, preference, or structured assessment. This is an automated grading method within model evaluation, rather than an independent source of truth. A judge can assess qualities such as relevance or instruction following, but its suitability must be established for each task.

## Why it matters

Open-ended outputs are expensive to review and often cannot be scored by exact matching. A validated judge can help screen regressions and prioritize human review. An unreliable judge can instead reward fluent mistakes and make model comparisons appear more objective than they are.

## When to use it

Use model judging when the criteria can be expressed clearly and representative human judgments are available for validation. Prefer deterministic checks for exact requirements. Keep expert review for consequential or ambiguous cases where the judge cannot establish correctness.

## How it works

1. Define separate rubric dimensions, rating anchors, and the evidence needed to judge each dimension.
2. Build a representative human-reviewed sample, preserving disagreements rather than assuming all ratings are certain.
3. Test the judge against that sample and inspect errors by task, language, length, and severity.
4. For pairwise comparisons, swap candidate order and check sensitivity to verbosity or stylistic cues.
5. Version the judge model, rubric, and configuration, then retain human audits and revalidate after changes.

## A practical example

A team compares two meeting-summary prompts. A judge scores whether decisions and owners are supported by the transcript. Human reviewers find that it favors longer summaries even when they add unsupported details. The team revises the rubric and keeps unsupported-claim review as a separate check before using the scores to guide prompt selection.

## Where it breaks

Judges can exhibit position, verbosity, and self-preference biases. They may share failures with the evaluated model or lack the expertise and context to assess correctness. Agreement with one reviewer sample does not establish reliability on new tasks, and explanations can sound persuasive even when ratings are wrong.

## Common mistakes

- Using an unvalidated judge score as the sole release gate.
- Giving the judge no source evidence while asking it to verify factual support.
- Changing the judge between model comparisons without rerunning the baseline.
- Collapsing distinct quality dimensions into an unexplained overall score.

## Visualization

[View the visualization on the website](<https://wiki.soroush.uk/topic/llm-as-a-judge/#visualization-llm-as-a-judge>)

## Sources

1. [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](<https://arxiv.org/abs/2306.05685>) — NeurIPS · Peer-reviewed research
2. [G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment](<https://aclanthology.org/2023.emnlp-main.153/>) — Association for Computational Linguistics · Peer-reviewed research

## Related topics

- [Model Evaluation](<https://wiki.soroush.uk/topic/model-evaluation/>)
- [Data Labeling](<https://wiki.soroush.uk/topic/data-labeling/>)
- [AI Hallucination](<https://wiki.soroush.uk/topic/hallucination/>)
