ClaudeSuperPower
SubmitLog in

output-eval-validate-judge

Skill

Validate LLM judges against human labels using TPR/TNR metrics and train/dev/test splits. Use after writing a judge prompt to verify it agrees with human judgment.

0Machine Learning

Install

git clone https://github.com/growthxai/output.git ~/.claude/skills/output-eval-validate-judge

What is output-eval-validate-judge?

Validate LLM judges against human labels using TPR/TNR metrics and train/dev/test splits. Use after writing a judge prompt to verify it agrees with human judgment.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Run shell commands

Declares Bash

Create and modify files

Declares Write or Edit

~41 tokens of context used while enabled, before you invoke anything

All declared tools (4)
BashEditReadWrite

Documentation

README · ~7 min read

Validating LLM Judges

Overview

An LLM judge is only useful if it agrees with human judgment. This skill walks you through calibrating a judge against human-labeled data using True Positive Rate (TPR) and True Negative Rate (TNR) metrics. Do this before trusting any judgeVerdict(), judgeScore(), or judgeLabel() evaluator in your eval suite.

Prerequisites

  1. A judge .prompt file — Written following output-eval-judge-prompt
  2. ~100 human-labeled traces — With binary pass/fail labels for the failure mode this judge targets. Aim for ~50 pass and ~50 fail. Minimum: 20 pass and 20 fail.
  3. Labels stored in dataset YAML — Each dataset has ground_truth.evals.<evaluator_name>.verdict: pass or fail

This process applies only to LLM-based judges. For code-based Verdict.* evaluators, write unit tests instead.

Step 1: Create Data Splits

Split your labeled datasets into three groups:

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.