Analytics and experiments

Every platform gives creators a dashboard full of numbers: views, impressions, click-through rates, retention graphs, traffic sources, demographics, revenue per thousand views. Used well, analytics show what the audience values and where content loses people, and turn guesswork into learning. Used badly, they become a source of anxiety, or lead creators to chase noise. Lesson 9 introduced experiments as part of a first-year plan. This lesson goes deeper: which metrics matter at each stage of the viewer journey, how to read retention graphs and traffic sources, how to judge whether a change really worked given the randomness in creator results, how to run simple experiments on titles, thumbnails and formats, and how to track the business metrics that matter for income.

A funnel of metrics

Analytics make most sense as a funnel that follows a viewer from first seeing a piece of content to becoming a fan or customer (Lesson 2):

Diagnosing a problem means finding where the funnel leaks. Low impressions suggest the platform is not testing the content widely, perhaps because of weak signals from the first viewers or a topic with little demand. Many impressions but low CTR point to packaging. Good CTR but low retention points to content that does not fulfil its promise or loses pace. Strong views with few follows suggest one-off interest rather than a channel people want more of.

Reading retention graphs

The retention graph, the share of viewers still watching at each moment, is the most informative chart available to a video creator. Patterns to look for:

• The initial drop: how many leave in the first 30 seconds. A steep drop means the hook does not match the packaging or is too slow (Lessons 2 and 6).
• Gradual decline: normal; a flatter line means a more engaging body.
• Sudden dips: specific moments where many viewers leave, such as a long sponsor segment, a tangent, a technical problem or an unclear explanation.
• Bumps: moments viewers rewatch or skip to, such as a key result, which suggest what the audience values most and could come earlier.
• The end: a large drop before the true end often signals a long outro; viewers leave when they sense the content is over.

Comparing a video's curve with the platform's benchmark for similar videos, and with the creator's own previous videos, shows whether a moment is unusually weak.

Traffic sources and audiences

Traffic source reports show where views come from: recommendations on the home page or feed, suggested videos, search, external sites, playlists, notifications to subscribers, and direct links. Each tells a different story. A video getting most views from search is likely evergreen and will keep earning; one driven by browse features depends on continued recommendation; external traffic shows sharing outside the platform. Audience reports show new versus returning viewers, demographics, geography, devices and when the audience is online. A channel where returning viewers make up a growing share is building loyalty; one dominated by new viewers who never return may be getting reach without relationship.

Noise, variation and small samples

Creator results are highly variable (Lesson 4): the same creator's similar videos can differ in views by ten times or more because of timing, competition, topic interest and the compounding of test waves. This variability makes it easy to see patterns that are not there. A thumbnail change followed by a rise in views may be coincidence; a "bad" week may be normal fluctuation. Practical safeguards:

• Compare averages over several pieces, not single results, and look at medians, which are less affected by one viral hit.
• Compare like with like: similar topics, lengths and times since publication (for example views in the first 7 days).
• Use rates (CTR, retention, conversion) rather than raw totals where possible, since rates depend less on how widely a piece was shown.
• Be wary of explaining every rise and fall; focus on persistent trends over months.

Running experiments

An experiment changes one thing deliberately and measures the effect against a comparison. Creators can experiment with packaging, hooks, length, formats, topics, posting schedules and calls to action. Good experiment practice:

• State a hypothesis in advance: "Opening with the finished result will raise 30-second retention from 65% to 75%."
• Change one variable at a time, keeping others as similar as possible.
• Decide the metric and how long to measure before looking at results.
• Run the test over several pieces, or use built-in testing tools, to reduce noise.
• Record results in a simple log, including failures, so that lessons accumulate.

Some platforms offer split tests: YouTube's "Test and compare" feature shows different thumbnails (and in some cases titles) to different viewers and picks the one with the most watch time, and several platforms allow testing of posts or ads with separate audiences. These randomized tests are much more reliable than before-and-after comparisons, because both versions face the same conditions at the same time.

Is the difference real?

With a randomized test, a simple statistical check shows whether a difference is larger than chance would produce. For two click-through rates \(p_1\) and \(p_2\) from \(n_1\) and \(n_2\) impressions, with pooled rate \(\bar p\), the z statistic is

and a value of \(|z|\) above about 2 suggests the difference is unlikely to be due to chance alone (roughly 95% confidence).

Business metrics

For creators who earn money, a few business metrics connect content to income (Lesson 5):

• RPM (revenue per mille): total revenue per 1,000 views, after the platform's share; it differs from the advertiser-facing CPM.
• Revenue per piece and per hour of work: helps decide which formats and projects are worth the effort.
• Conversion rates: of viewers to email subscribers, of subscribers to customers or members.
• Churn: the share of members or subscribers who cancel each month; reducing churn is often easier than finding new members.
• Customer lifetime value: average revenue per customer over the relationship, which shows how much can be spent to acquire one.

Using analytics without being used by them

Constant checking of real-time numbers is a recognized source of stress for creators (Lesson 7) and rarely improves decisions. Healthier habits include reviewing analytics at set times (for example 48 hours and 7 days after publishing, and a monthly overview), focusing on a few metrics tied to current goals, keeping notes on what was learned rather than just numbers, and remembering that metrics are proxies: a video that helps a small number of people a great deal may matter more than one that performs well briefly. Qualitative feedback, the comments and messages that explain why people watch, complements the numbers.

A worked diagnosis

Suppose a creator notices that a new series of videos is getting fewer views than her usual uploads. Rather than guessing, she works down the funnel. Impressions are similar to her other videos, so the platform is still showing them. Click-through rate is 3.1 percent against her usual 5 percent, so fewer people choose to watch. Average percentage viewed is slightly higher than usual, so the people who do click are satisfied. The problem is the packaging, not the content: titles and thumbnails for the new series are not telling browsers what they will get.

She writes two new thumbnail and title options for each of the next three videos and uses the platform's built-in testing tool where available. One option names the specific problem the video solves; the other uses a bold image with few words. The problem-focused option wins on two of three videos, though only one difference is large enough to be confident about at the sample sizes involved. She adopts the problem-focused style, keeps watching click-through rate and records the decision in her experiment log so she can review it later. The whole process took a few hours and replaced a vague worry with a specific, testable fix.