ClaudeSuperPower
SubmitLog in

output-eval-judge-prompt

Skill

Design effective LLM judge .prompt files for evaluators. Use when creating judgeVerdict/judgeScore/judgeLabel prompts, or when existing judges produce unreliable results.

0Machine Learning

Install

git clone https://github.com/growthxai/output.git ~/.claude/skills/output-eval-judge-prompt

What is output-eval-judge-prompt?

Design effective LLM judge .prompt files for evaluators. Use when creating judgeVerdict/judgeScore/judgeLabel prompts, or when existing judges produce unreliable results.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Create and modify files

Declares Write or Edit

~43 tokens of context used while enabled, before you invoke anything

All declared tools (3)
EditReadWrite

Documentation

README · ~8 min read

Designing LLM Judge Prompts

Overview

An LLM judge evaluates workflow output for a single, specific failure mode identified during error analysis. This skill covers how to design the .prompt file that powers judgeVerdict(), judgeScore(), or judgeLabel() calls. For the file format basics, see output-dev-prompt-file. For error analysis, see output-eval-error-analysis.

Prerequisites

Before writing a judge prompt:

  1. Error analysis is complete — You have identified the specific failure mode this judge targets (from output-eval-error-analysis)
  2. 20+ labeled examples — At least 20 pass and 20 fail traces for this failure mode, with ground_truth labels in dataset YAML files
  3. Code-based check ruled out — Confirmed that Verdict.* helpers (contains, matches, gte, etc.) cannot reliably detect this failure

The Four Components

Every effective judge prompt has exactly four components.

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.