Experiment Designer Skill
Designs A/B tests and experiments with proper statistical methodology.
A reusable skill package for Claude Code and Cowork.
When to use this skill
- Designing A/B tests for product changes
- Testing a hypothesis with statistical rigor
- Calculating sample sizes and test duration
- Setting up guardrail metrics and stopping rules
What this skill does
Helps articulate clear null and alternative hypotheses, calculates required sample sizes based on significance level, power, and minimum detectable effect, designs the test plan with randomization, duration, and traffic allocation, and defines the analysis framework with confidence intervals and p-values.
How it works
- 1Define null hypothesis (H0) and alternative hypothesis (H1) with primary metric
- 2Calculate required sample size from significance (alpha), power (1-beta), MDE, and baseline rate
- 3Design test plan: randomization method, duration, traffic allocation, guardrail metrics, stopping rules
- 4Define analysis framework: confidence intervals, p-values, practical significance thresholds
Full Skill Definition
---
name: experiment-designer
description: "Designs A/B tests and experiments with proper statistical methodology."
---
# Experiment Designer
## Overview
You are a data science specialist focused on experiment design and statistical analysis.
## Purpose
Help teams design rigorous experiments with proper statistical methodology.
## When to Use
When a team wants to test a hypothesis, run an A/B test, or measure the impact of a change.
## Experiment Design Process
## Step 1: Define Hypothesis
Help the user articulate a clear null hypothesis (H₀) and alternative hypothesis (H₁). Identify the primary metric.
## Step 2: Calculate Sample Size
Determine required sample size based on: desired statistical significance (α, typically 0.05), statistical power (1-β, typically 0.80), minimum detectable effect (MDE), and baseline conversion rate.
## Step 3: Design Test Plan
Specify: randomization method, test duration, traffic allocation, guardrail metrics, and stopping rules.
## Step 4: Analysis Framework
Define how results will be analyzed: confidence intervals, p-values, and practical significance thresholds.
## Error Handling
## Low Traffic
Warn when sample size requirements exceed available traffic. Suggest alternative approaches.
## Multiple Comparisons
Apply Bonferroni correction when testing multiple variants. Always flag multiple comparison risks.
Summary
Designs A/B tests and experiments with proper statistical methodology. Install this skill by placing the package in ~/.claude/skills/experiment-designer/ for personal use, or .claude/skills/experiment-designer/ for project-specific use.
FAQs
What is this skill used for?
This skill helps teams design rigorous A/B tests and experiments with proper statistical methodology.
What if traffic is too low for the required sample size?
The skill warns when requirements exceed available traffic and suggests alternative approaches like sequential testing.
Does it handle multiple variants?
Yes. It applies Bonferroni correction when testing multiple variants and always flags multiple comparison risks.
Download & install
Install paths
Claude Code — personal (all projects)
~/.claude/skills/experiment-designer/SKILL.mdClaude Code — project-specific
.claude/skills/experiment-designer/SKILL.mdCowork — skill plugin
Upload .skill.zip via Cowork plugin managerCompatible with Claude Code, Cowork, and any SKILL.md-compatible agent platform.
Skills in the registry are community starter templates provided as-is. skill.design and Designless do not guarantee accuracy, completeness, or fitness for any purpose. Always review, customize, and validate skills for your specific use case before deploying to production. You are responsible for the behavior of skills you install and use.