ClaudeSuperPower
SubmitLog in

diagnosing-experiment-results

Skill

Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers empty / 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-S

0Analytics

Install

git clone https://github.com/PostHog/ai-plugin.git ~/.claude/skills/diagnosing-experiment-results

What is diagnosing-experiment-results?

Diagnoses bias, anomalies, and strange-looking results on a specific PostHog experiment. Covers empty / 0-exposure experiments, sample ratio mismatch, identity fragmentation, multi-variant exposure, uneven-split exclusion bias, significance traps (peeking, A/A, Bayesian vs Frequentist), PostHog-vs-SQL discrepancies, and surprises after mid-run edits. Symptom-driven dispatch to the right diagnostic.\nTRIGGER when: user asks 'is my experiment biased?' or 'why 0 exposures?', references the bias banner, says a variant looks strange / wrong / off, sees significance flipping, notices PostHog numbers disagreeing with their SQL, sees an A/A test showing significance, or reports surprises after mid-run edits.\nDO NOT TRIGGER when: creating a new experiment (use creating-experiments), only configuring rollout (use configuring-experiment-rollout) or metrics (use configuring-experiment-analytics), or only asking lifecycle questions (use managing-experiment-lifecycle).

What this can do

Capabilities declared in this component's own frontmatter — not inferred.

Inherit all session tools

Declares no tool restrictions — inherits every session tool

~243 tokens of context used while enabled, before you invoke anything

Documentation

README · ~7 min read

Diagnosing experiment results

This skill answers: My PostHog experiment results look wrong, biased, or empty — what's going on?

Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.

Each diagnostic in the reference files is tagged [HIGH], [MEDIUM], or [LOW] based on how strongly it's verified — [HIGH] is verified directly in PostHog code, [MEDIUM] is partially or team-source verified, [LOW] describes SDK/external behavior that wasn't verified here. Treat [LOW] items as hypotheses to test, not facts to assert.

Step 1 — Resolve the experiment

If the user refers to an experiment by name or description, load the finding-experiments skill first to

Reviews

Log in to leave a review.

No reviews yet — be the first.

Explore related

Other things in this space — across every part of the ecosystem, not just skills.

Skillssimilar to this one

All skills →

find-test-content

Use this when you need to find existing pages that already use a specific block in an AEM Edge Delivery Services project, for example to locate test content or examples during block development. Covers reporting page URLs with occurrence counts and block variants. This searches existing content; to

151 stars

code-assessment

[BETA] Detect, review, and fix code-quality and correctness issues in an AEM as a Cloud Service project — locally, with no external services or network calls. Use whenever a user wants to check, review, assess, audit, scan, modernize, upgrade, or fix AEM Java, Sling Models, OSGi, or Maven code — for

151 stars

cja-dimension-analysis

Comprehensive dimension analysis and reporting for CJA. Use this skill whenever the user wants to analyze one or more dimensions — including cardinality, distribution/skew, trends, anomalies, data quality errors, comparisons, and forecasting. Also trigger when someone asks "what are the top values f

151 stars