ClaudeSuperPower
SubmitLog in

exploring-ai-failures

Skill

Find where an AI/LLM application is failing in production and surface the failure patterns, working from real traces. Use when someone wants to understand what's going wrong with an AI feature, find and categorize failure modes, triage errors, or investigate quality issues (wrong answers, ignored in

0Machine Learning

Install

git clone https://github.com/PostHog/ai-plugin.git ~/.claude/skills/exploring-ai-failures

What is exploring-ai-failures?

Find where an AI/LLM application is failing in production and surface the failure patterns, working from real traces. Use when someone wants to understand what's going wrong with an AI feature, find and categorize failure modes, triage errors, or investigate quality issues (wrong answers, ignored instructions, hallucinations, tool misuse) — "what's failing in my agent", "surface error patterns", "why are the responses bad", "find the common failure modes", "what should I fix next". Covers scoping to one use case, finding failing traces by whichever signal fits the context (code errors, metric outliers, trace-type slices, manual review, existing-eval spikes, clustering), and reading them into a ranked failure taxonomy.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~182 tokens of context used while enabled, before you invoke anything

Documentation

README · ~7 min read

Exploring AI failures

The highest-value thing you can do with production AI traffic is look at where it fails and name the patterns. The catch: most failures are silent. The model returns a clean response — HTTP 200, no exception — that is wrong, off-topic, ignores an instruction, or misuses a tool. Those never raise an error, and they're usually the failures worth caring about.

So this skill is about finding failures (loud and silent), reading them, and grouping them into a ranked set of failure modes you can act on: fix a prompt, file a bug, prioritize work, or turn the top mode into an automatic eval (creating-online-evaluations).

Everything below serves one irreducible activity: reading real traces. The queries only tell you which traces to open — they are never the answer. If you report a list of problems without having

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.