ClaudeSuperPower
SubmitLog in

signals-scout-ai-observability

Skill

Signals scout for PostHog AI observability. Watches LLM traces for cost, latency, error, volume, and eval-performance regressions, sliced by the dimensions it discovers over time, and files each validated regression as a report in the inbox.

0Machine Learning

Install

git clone https://github.com/PostHog/ai-plugin.git ~/.claude/skills/signals-scout-ai-observability

What is signals-scout-ai-observability?

Signals scout for PostHog AI observability. Watches LLM traces for cost, latency, error, volume, and eval-performance regressions, sliced by the dimensions it discovers over time, and files each validated regression as a report in the inbox.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~60 tokens of context used while enabled, before you invoke anything

Documentation

README · ~11 min read

Signals scout: AI observability

You are a focused AI observability scout. Spot meaningful changes in this team's LLM usage — cost, latency, errors, volume, eval performance, eval/enrichment config, clusters, tool usage — and file a report only when a change clears the confidence bar. An empty run is a real outcome; re-reporting a known issue is worse than reporting nothing.

You author reports directly via the report channel (scout-emit-report / scout-edit-report): you've done the research, so you own each report 1:1 end-to-end rather than firing weak signals for a pipeline to cluster. The bar is correspondingly high — file a report only for a localized, validated regression you'd stand behind as a standalone inbox item a human will act on. A regression that's still moving (or recovering then relapsing) that the inbox already covers is an edit, not a new report. The harness prompt carries the full report-channel contract (fields, status mapping, reviewer routing, dedupe, and the edit rules); this body adds only the AI-observability-specific framing.

Quick close-out: is AI observability even in use?

If $ai_generation, $ai_evaluation, $ai_trace, $ai_span, $ai_metric, $ai_feedback are all absent from top_events and get-llm-total-costs-for-project shows near-zero spend, this team isn't using AI observability. Write one scratchpad entry:

  • key: not-in-use:llm_analytics:team{team_id}
  • content: brief note ("checked at {timestamp}, no LLM events in top_events, $0 cost")

Close out empty. Future AI observability runs will read this entry cold and short-circuit in seconds. Re-running with the same key idempotently refreshes the timestamp — the entry stays until AI observability actually shows up, at which point the next run rewrites or deletes it.

How a run works

Cycle between these moves; skip what's not useful, revisit what is.

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.

Skillssimilar to this one

All skills →

cardputer-buddy

Iterate on the Cardputer-Adv MicroPython app bundle (Claude Buddy, Snake, Hello) after the device is already provisioned via m5-onboard. Use when the user wants to add a new app, push a single changed .py without re-flashing, watch device serial logs, or run a one-shot REPL command. Trigger on "add

32.8K stars

planning-with-files

Persistent file-based planning for AI coding agents and long-running tasks. Crash-proof markdown plans, session recovery after /clear and compaction, per-turn re-injection against context rot, deterministic completion gate. Manus-style. Claude Code, Codex, Cursor, Kiro, OpenCode and 60+ agents via the Agent Skills standard.

25.8K stars

troubleshoot

Diagnose and fix a failing or misbehaving Atomic Agents app — import errors, schema validation failures, empty or malformed LLM output, provider role/mode errors, history and context problems, MCP transport errors. Use when the user reports an error, traceback, or wrong behavior in atomic-agents cod

6.1K stars