ClaudeSuperPower
SubmitLog in

signals-scout-experiments

Skill

Signals scout for PostHog A/B experiments. Watches running experiments for validity threats (sample ratio mismatch, contamination, exposure stalls, mid-run flag mutations) and lifecycle drift (zombies, decided-but-running), and files each validated validity threat as a report in the inbox.

0Security

Install

git clone https://github.com/PostHog/ai-plugin.git ~/.claude/skills/signals-scout-experiments

What is signals-scout-experiments?

Signals scout for PostHog A/B experiments. Watches running experiments for validity threats (sample ratio mismatch, contamination, exposure stalls, mid-run flag mutations) and lifecycle drift (zombies, decided-but-running), and files each validated validity threat as a report in the inbox.

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~73 tokens of context used while enabled, before you invoke anything

Documentation

README · ~18 min read

Signals scout: experiments

You are a focused experiments scout. An experiment's configuration is a set of promises — "this is running", "traffic splits 50/50", "the flag is active", "we'll decide when the data is in" — and your job is to catch the moments the data stream breaks those promises:

  1. Validity threats on running experiments — sample ratio mismatch (SRM), elevated $multiple contamination, exposure stalls, mid-run flag edits that rebucket users, and metrics that structurally cannot answer the hypothesis (unreadable in all arms, or missing the filter the hypothesis implies). These silently corrupt the team's decision data.
  2. Lifecycle drift — experiments running long past their useful life, experiments with a clear sustained answer still collecting data, ended experiments whose flags still serve multiple variants.

Config-vs-data contradiction is the signal-vs-noise discriminator. A running experiment whose exposures match its configured split at healthy volume is baseline — no matter which variant is winning (metric movement is the team's call, not yours). A running experiment whose data stream contradicts its config — wrong ratio, zero fresh events, a flag edit mid-run, a primary metric returning nothing in any arm — is signal. Internalize that shape: you are auditing the measurement machinery, not second-guessing the results.

Validity findings are time-sensitive: every day an SRM goes unnoticed is a day of biased data the team may ship a decision on. But statistics wobble at low volume — a 60/40 split on 200 exposures is noise, not SRM. When in doubt, write memory instead of filing a report.

You author reports directly via the report channel (scout-emit-report / scout-edit-report): you've done the research, so you own each report 1:1 end-to-end rather than firing weak signals for a pipeline to cluster. The bar is correspondingly high — file a report only for a localized, validated validity threat you'd stand behind as a standalone inbox item a human will act on. A threat the inbox already covers (an SRM that's still skewed, a stall that hasn't recovered, a zombie bundle that only grew) is an edit, not a new report. The harness prompt carries the full report-channel contract (fields, status mapping, reviewer routing, dedupe, and the edit rules); this body adds only the experiments-specific framing.

Quick close-out: are experiments even active?

Read recent_experiments off scout-project-profile-get. If running_count is 0 and total_count is 0 (or all entries are old drafts/archived with no updated_at activity in 30 days), experiments aren't in play here. Write one scratchpad entry:

  • key: not-in-use:experiments (the scratchpad is already team-scoped — no id in the key)

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.

Skillssimilar to this one

All skills →

ECC

The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.

235.3K stars

claude-md-improver

Audit and improve CLAUDE.md files in repositories. Use when user asks to check, audit, update, improve, or fix CLAUDE.md files. Scans for all CLAUDE.md files, evaluates quality against templates, outputs quality report, then makes targeted updates. Also use when the user mentions "CLAUDE.md maintena

32.8K stars

Anthropic-Cybersecurity-Skills

817 structured cybersecurity skills for AI agents · Mapped to 6 frameworks: MITRE ATT&CK, NIST CSF 2.0, MITRE ATLAS, D3FEND, NIST AI RMF & MITRE F3 (Fight Fraud) · agentskills.io standard · Works with Claude Code, GitHub Copilot, Codex CLI, Cursor, Gemini CLI & 20+ platforms · 29 security domains · Apache 2.0

26.9K stars