Experiment Design for Creators

Why This Lesson Matters

Sequence you are in:

• Previous: Analytics Deep Dive for LIVE Creators — reading numbers with context, not panic
• This lesson: Experiment Design for Creators — clean variables, sample humility, real documentation
• Next: Scheduling as Strategy — turning one experiment type into a full cadence decision

Advanced Creator taught you not to thrash. Growth Mastery now teaches you to design a test that actually produces a trustworthy answer. Every later Growth Mastery lesson — scheduling, discovery, algorithm-durable habits — assumes you can run a clean test instead of a hopeful guess. Sloppy experiment design is how creators end up "proving" things that were never true.

Learning Objectives

By the end of this lesson, you will be able to:

• Separate the variable you changed from the outcome measure you are watching
• Establish a baseline before you claim any result means something
• Apply sample-size humility so one good or bad session does not become a myth
• Write a kill criterion that covers both safety and statistical honesty
• Document a full Experiment Log entry with a real conclusion, not just a verdict
• Decide when a result is strong enough to act on and when it needs more sessions

Estimated Study Time

• Study and design: about 30 minutes to complete the Experiment Design Sheet
• LIVE Mission: one full LIVE session that runs the designed test cleanly
• Documentation: 10–15 minutes immediately after LIVE to complete the Conclusion Log entry

Prerequisites

Complete Analytics Deep Dive for LIVE Creators (`analytics-deep-dive-for-live-creators`) and Growth Experiments That Don't Wreck Your Show (`growth-experiments-that-dont-wreck-your-show`).

You should already have:

• Comfort reading your own session numbers without panic or vanity framing
• The habit of changing one variable at a time with a written kill rule
• A brand off-limits list that experiments never override

Main Lesson

From "one variable" to a clean test

A clean test has four named parts before session one:

1. Variable — the single thing you are changing on purpose 2. Outcome measure — the specific, observable number or behavior you are watching 3. Baseline — what that measure normally looks like without the change 4. Stopping rule — the point at which you stop and read the result, win or lose

If you can only name the variable, you have half a design. Most creator experiments die at the baseline step — they change something and then compare the result to a feeling instead of a number they already had.

Variable and outcome measure — keep them separate

The variable is what you do differently. The outcome measure is what tells you whether it worked. Confusing the two is the single most common design error.

Bad pairing: "I'll try a new hook and see if it feels good." Neither the outcome nor the measure is specific.

Clean pairing: "Variable: open with a named promise in the first fifteen seconds. Outcome measure: average minutes watched in the first ten minutes, versus my prior five-session baseline."

Pick one primary outcome measure. Note secondary observations if you want, but only the primary measure decides the result. Watching five numbers at once turns an experiment into a story you can bend to whatever conclusion you already wanted.

Sample-size humility

One session is not a trend. Even six sessions is a small sample when your normal week already swings for reasons that have nothing to do with your variable — a platform glitch, a bad sleep night, a holiday, a lucky push.

Sample-size humility means you state, before you start, how many sessions you need before you will trust the result at all — and you say out loud what a single strong or weak session actually proves: almost nothing on its own.

Practical guide for creator-scale testing:

• One session: a data point, not a result.
• Three to four sessions: an early read — catches obvious failures, too small for success.
• Five to eight sessions across two to three weeks: enough for an honest keep/adapt/kill call.

Write your minimum sample size on the Sample-Size Humility Card before session one. That card is what stops you from mistaking one great night for a finding.

Kill criteria: safety kill and statistical kill

AC-05 taught you a single kill rule. Here you separate two kinds:

Safety kill — stop immediately, no matter the sample size, if the variable breaks platform safety, brand off-limits, or your wellbeing.

Statistical kill — stop at your committed sample size (or earlier if every session so far is unmistakably worse) if the outcome measure stays below baseline with no upward movement. A calm, planned stop — not a panic reaction to one bad night.

Write both before you start. A design without a statistical kill tends to run forever, because nobody wants to admit the number never moved.

Documenting the Experiment Log properly

A result nobody can read later is not evidence — it is a memory that will drift. Each entry should record: date, variable present (yes/no), the outcome measure's actual number, baseline for comparison, and one honest note about anything unusual (tech issue, low sleep, holiday). At the committed sample size, write a short conclusion: what changed, by how much, against what baseline, and your decision — keep, adapt, or kill — in one sentence a stranger could understand.

Capstone and Honors Lab connection

Your completed Experiment Design Sheet and Conclusion Log become primary Capstone dossier evidence — reviewers want to see design rigor, not just a decision. The 30-Day Pro Sprint expects at least one experiment with a real baseline and sample size, not a vibes retrospective. Optional Honors Lab review may ask whether your baseline was real and your sample size was met before you called a verdict — never a certificate gate.

Examples

Example 1 — Hook variable, done cleanly. Variable: state the session's promise in the first fifteen seconds. Outcome measure: average minutes watched in minutes 0–10, baseline 6.2 minutes from the last five sessions. Sample size: six sessions. Result: 8.1 minutes average, trending up from session two. Conclusion: keep.

Example 2 — CTA variable with a statistical kill. Variable: close every session naming the next LIVE day and time. Outcome measure: named returners at the start of the following session, baseline of 4. Sample size: five sessions. After three, no movement above baseline. Conclusion at planned stop: adapt — the line may be too generic; rewrite before retesting rather than declaring CTAs useless.

Example 3 — A one-session illusion caught by humility. Variable: new opening segment. Session one spikes far above baseline; the creator wants to declare victory immediately. The Sample-Size Humility Card says wait for six. Sessions two through five return to baseline. Conclusion: the spike was likely an outside factor, not the variable — kill and investigate separately.

Real Creator Scenarios

Scenario A — "It obviously worked, I felt it." Action: check the Experiment Design Sheet. If the outcome measure and baseline were not written down first, this is a feeling, not a finding. Rerun properly before changing anything permanently.

Scenario B — "I'm two sessions in and it's already worse." Action: check for a safety kill first. If it's only a statistical dip, hold to your committed sample size unless every single session so far is clearly worse — then you may invoke an early statistical kill and say so honestly in the log.

Scenario C — "The result was small — barely above baseline." Action: a small, consistent improvement across a full sample is still a real result. Write "keep, modest effect" instead of forcing it into a bigger story than the numbers support.

From Brad's Experience

Pro Tips

• Write the outcome measure as a number, not a feeling.
• Pull a real baseline from your last three to five sessions before you start.
• Commit to a sample size before session one, in writing.
• Keep safety kill and statistical kill as two separate lines.
• Never let one outlier session end the test early unless it is a safety issue.
• Log every session, including the boring or messy ones.
• Write the conclusion so a stranger could understand it without asking follow-up questions.
• File the completed sheet and log for the Capstone dossier.

Common Beginner Mistakes

• No baseline before starting. Fix: pull your last few sessions' numbers first.
• Watching five outcome measures at once. Fix: pick one primary measure before you start.
• Declaring victory after one great session. Fix: hold to your committed sample size.
• No statistical kill, only a safety kill. Fix: write both before session one.
• Vague conclusions like "it worked." Fix: state the number, the baseline, and the decision.
• Changing the outcome measure mid-test to match what happened. Fix: lock it before you start; that is the whole point of a measure.
• Treating a small improvement as failure because it wasn't dramatic. Fix: a real, consistent gain is still a win.

Reality Check

Most honest experiments end with modest, unglamorous conclusions — a small gain, a clear miss, or "not enough signal yet, extend the sample." That is a professional result, not a disappointing one.

If your instinct is to skip the baseline or sample-size commitment because "you already know it worked," that instinct is exactly what this lesson exists to slow down. Confidence without a baseline is just certainty borrowed from nowhere.

Summary

A clean creator experiment names a variable and a single outcome measure, compares against a real baseline, commits to a sample size before starting, separates safety kill from statistical kill, and ends in a documented, readable conclusion. This is the difference between testing and storytelling.

LIVE Mission

Mission: Clean Experiment Session

1. Complete the Experiment Design Sheet — variable, outcome measure, baseline (from your last three to five sessions), committed sample size, safety kill, and statistical kill. 2. Fill out the Sample-Size Humility Card before you go LIVE. 3. Run a LIVE session that executes the variable exactly as designed. 4. Immediately after, log the session's outcome number in the Experiment Log — do not wait until tomorrow.

Success is a completed, honest design and one clean logged session — not a dramatic result.

Downloads

• Clean Experiment Design Sheet — variable, outcome measure, baseline, sample size, dual kill criteria
• Sample-Size Humility Card — one-session vs early-read vs committed-sample guidance
• Experiment Conclusion Log — session rows plus a final one-sentence decision field

Quiz

Take the interactive lesson quiz on this page (70% to pass). It checks whether you can design a clean test — not whether you can memorize statistics vocabulary.

Key Takeaways

• A clean design names a variable, an outcome measure, a baseline, and a sample size before session one
• One session is a data point, not a result
• Safety kill and statistical kill are two different lines, both written in advance
• Never change the outcome measure mid-test to match what happened
• A modest, consistent result is still a real result
• Documentation should be readable by a stranger, not just memorable to you
• The Experiment Design Sheet and Conclusion Log feed the Capstone dossier
• Honors Labs may audit baseline and sample-size honesty — never as a gate

Before You Move On

☐ Finished reading this lesson

☐ Completed the Experiment Design Sheet with a real baseline

☐ Set a committed sample size and wrote both kill criteria

☐ Passed the Lesson Quiz (70%+)

☐ Completed the Clean Experiment Session LIVE Mission

☐ Logged the session's outcome number immediately after LIVE

☐ Scheduled the remaining sessions needed to reach your committed sample size

☐ Filed the design sheet and log for Capstone

Next Lesson Preview

Next up: Scheduling as Strategy. You will apply this same rigor to one of the biggest growth levers you control — when you actually go LIVE. Expect a real test of your assumed "best time," an honest look at when your specific audience is actually online, and a schedule you can sustain past week three.