Interpreting results and avoiding traps

Running an experiment is only half the work. The other half is interpreting the result correctly, and this is where many creators go wrong, even with good data. Human minds are built to find patterns and stories, and creator data offers endless opportunities to find ones that are not there. This lesson describes the most common traps in interpreting creator data: regression to the mean, survivorship bias, seasonality, novelty effects, algorithm and platform changes, Simpson's paradox, and confusing correlation with causation. For each, it gives a way to recognize it and a way to guard against it.

Regression to the mean

Regression to the mean is the tendency for unusually high or low results to be followed by results closer to the average, simply because extreme results usually include a large dose of luck. If a video performs far above your typical level, the next one will probably perform worse, not because you did anything wrong, but because the first result was partly lucky. Likewise, after a very weak video, the next one will probably do better, whatever you change.

This creates a dangerous illusion. Creators tend to make changes after a disappointing video, and the next video usually improves, so the change appears to have worked. Creators also tend to keep doing the same thing after a hit, and the next video usually does worse, so they conclude the audience is tiring. Both conclusions can be pure regression to the mean.

The guard against regression to the mean is to compare results against a stable baseline, such as the median of many videos, rather than against the previous result, and to use controlled designs whenever possible.

Survivorship bias

Survivorship bias occurs when you study only the examples that succeeded and ignore the ones that failed, which leads to false conclusions about what causes success. Creators encounter it constantly. Advice from very successful creators, lists of features shared by viral videos, and case studies of channels that grew quickly all focus on survivors. If thousands of creators used the same tactics and failed, those failures are invisible.

The guard is to compare successes with the full population, including failures, and to treat lists of winners' habits as hypotheses to test, not as proven recipes.

Seasonality and calendar effects

Viewing behaviour changes across the year, the week and the day. Gift guides surge in November and December; fitness content in January; back-to-school content in August and September; tax content in the spring, with Canada's April 30 filing deadline and the US April 15 deadline creating slightly different peaks. Weekends differ from weekdays, and holidays such as Canadian Thanksgiving in October and US Thanksgiving in November shift viewing patterns. Any comparison across periods must account for these effects, as lesson 5 showed with seasonal adjustment.

Novelty and fatigue effects

When something changes, audiences often react to the change itself. A new thumbnail style may attract extra clicks at first simply because it looks different from what returning viewers are used to, and the effect may fade as it becomes familiar. This is a novelty effect. The opposite, a change effect or temporary dip, can happen when returning viewers are briefly confused by a new look. Over longer periods, audiences can also tire of a repeated format, called fatigue. Both mean that short tests can overstate or understate long-term effects.

The guard is to run tests long enough to see past initial reactions, to compare new and returning viewers separately where possible, and to re-check important changes after a few months.

Algorithm and platform changes

Platforms change their recommendation systems, interfaces and metric definitions frequently, often without detailed announcements. A sudden change in your numbers may reflect a platform update rather than anything you did. Lesson 3 showed how YouTube's 2025 change in Shorts view counting could make growth appear much larger than it was. Interface changes, such as a new placement for Shorts in the app or a redesign of the home feed, can shift traffic sources overnight.

Guards include keeping a log of known platform changes, watching creator communities and official platform blogs for announcements, comparing your trends with those of similar creators where possible, and using metrics such as watch time or engaged views that are less sensitive to counting changes.

Simpson's paradox

Simpson's paradox occurs when a trend that appears in several groups reverses or disappears when the groups are combined. It arises when the groups differ in size and their mix changes. Lesson 3 showed a version with click-through rates by traffic source. It also appears when comparing formats or platforms.

The guard is to compare like with like: break results down by the factors that might differ between groups, such as topic type, traffic source or video length, before drawing conclusions.

Correlation is not causation

Two metrics that move together are correlated, but that does not mean one causes the other. Videos with more comments may get more views because the algorithm rewards engagement, or because popular topics generate both comments and views. Longer videos may correlate with higher revenue because they allow more ad breaks, or because the creator makes longer videos on topics advertisers value. To establish cause, you need an experiment or a careful design that rules out the other explanations.

A short interpretation checklist

Before acting on any result, ask these questions.

• Am I comparing against a stable baseline or just the last result?
• Am I looking only at successes?
• Could season, holidays or events explain this?
• Could this be a novelty effect that will fade?
• Did the platform change anything in this period?
• Would the result hold within subgroups, or is it a mix effect?
• Is this a correlation that needs an experiment to confirm cause?

No experiment is perfect, and creators must still make decisions with incomplete evidence. The goal is not certainty but calibrated confidence: acting boldly on strong evidence, cautiously on weak evidence, and always leaving room to learn more.

Writing up results honestly

How you record a result shapes what you and your team will believe later. A good write-up states the original hypothesis and prediction, the design, the result with its uncertainty, the traps you checked for, and the decision. It avoids overclaiming: "Retention at 30 seconds rose by about 4 points across eight videos; the 95 % interval was roughly 1 to 7 points; we adopted the change and will recheck in three months for novelty effects" is far more useful than "The new intro works." Honest write-ups make it easier to revisit decisions when new evidence arrives.