ClaudeSuperPower
SubmitLog in

feature-usage-feed

Skill

Set up an LLM-judge evaluation that extracts canonical use cases for a PostHog feature at scale and streams the results to a Slack channel as a live feed. Use when someone wants to understand how users are actually using a specific AI/LLM-powered feature in production — what they're investigating, w

0Machine Learning

Install

git clone https://github.com/PostHog/ai-plugin.git ~/.claude/skills/feature-usage-feed

What is feature-usage-feed?

Set up an LLM-judge evaluation that extracts canonical use cases for a PostHog feature at scale and streams the results to a Slack channel as a live feed. Use when someone wants to understand how users are actually using a specific AI/LLM-powered feature in production — what they're investigating, what questions they're trying to answer, and what patterns surface — without manually reading hundreds of traces. Assumes the feature emits `$ai_generation` and `$ai_evaluation` events with `$session_id` linkage to the trigger user's recording (the standard setup post the session-summary linkage PRs).

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~150 tokens of context used while enabled, before you invoke anything

Documentation

README · ~17 min read

Building a feature usage feed via LLM evals

Some PostHog features (group session summaries, single session summaries, replay AI search, error tracking AI debug, etc.) generate hundreds or thousands of LLM traces per week. Reading them by hand is not feasible. This skill covers the end-to-end pattern for turning that trace volume into a live Slack feed of canonical use cases — what users are actually doing with the feature.

The workflow is mixed, and leans UI. Trace inspection and filter discovery (steps 1-2) are MCP-driven. Eval creation, dry-running, and enabling (steps 4-5) are MCP-driven when posthog:llma-evaluation-* tools are exposed to your agent — but they often aren't, in which case fall back to the UI (Data pipeline → destinations for the alert is always UI). Each step flags its UI fallback. Expect to finish in the UI even when you start from chat.

When to use

  • "How are people actually using [feature X] in production?"
  • "Can we identify the canonical use cases for [feature X] so we can write better docs / prioritize improvements?"
  • "I want a Slack feed of representative usage examples without manually skimming traces."
  • "Set up a feed of use cases for [feature X] in #team-[area]-usage."

If the user just wants to debug a single trace or tune an existing eval, redirect to exploring-llm-traces or exploring-llm-evaluations instead.

Two filter patterns

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.