ClaudeSuperPower
SubmitLog in

exploring-llm-evaluations

Skill

Investigate AI observability evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and generate AI-powered summ

0Debugging

Install

git clone https://github.com/PostHog/ai-plugin.git ~/.claude/skills/exploring-llm-evaluations

What is exploring-llm-evaluations?

Investigate AI observability evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and generate AI-powered summaries for boolean pass/fail runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, inspect sentiment classifications, or manage the evaluation lifecycle.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~147 tokens of context used while enabled, before you invoke anything

Documentation

README · ~10 min read

Exploring AI observability evaluations

PostHog evaluations score $ai_generation events. Each evaluation is one of three types:

  • hog — deterministic Hog code that returns true/false (and optionally N/A). Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.
  • llm_judge — an LLM scores generations against a prompt you write. Best for subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.

Skillssimilar to this one

All skills →

cardputer-buddy

Iterate on the Cardputer-Adv MicroPython app bundle (Claude Buddy, Snake, Hello) after the device is already provisioned via m5-onboard. Use when the user wants to add a new app, push a single changed .py without re-flashing, watch device serial logs, or run a one-shot REPL command. Trigger on "add

32.8K stars

planning-with-files

Persistent file-based planning for AI coding agents and long-running tasks. Crash-proof markdown plans, session recovery after /clear and compaction, per-turn re-injection against context rot, deterministic completion gate. Manus-style. Claude Code, Codex, Cursor, Kiro, OpenCode and 60+ agents via the Agent Skills standard.

25.8K stars

troubleshoot

Diagnose and fix a failing or misbehaving Atomic Agents app — import errors, schema validation failures, empty or malformed LLM output, provider role/mode errors, history and context problems, MCP transport errors. Use when the user reports an error, traceback, or wrong behavior in atomic-agents cod

6.1K stars