Product Metrics

Why metric fluency matters

Product DS loops always test product sense — the ability to reason about a product through its metrics. The questions sound conversational:

• "How would you measure success for [feature]?"
• "What's the right north star metric for [product]?"
• "Metric X dropped 5% — what do you do?"
• "We're launching [thing]; what's the metric set you'd track?"

These are easy to answer badly and hard to answer well. Bad answers list metrics generically. Good answers commit to a hierarchy, defend the primary, anticipate failure modes, and tell the interviewer what you'd ignore on purpose.

North star metrics

A north star is the one metric that, if it moves consistently up, the company is winning. It has to be:

• Tied to user value — moving it should require giving users something they want, not gaming.
• Movable — the team can affect it within a meaningful time horizon. "Lifetime customer happiness" is a poor north star; "weekly active creators publishing > 1 video" might work.
• Sensitive — small product changes should produce visible movement in a reasonable window.
• Resistant to gaming — chosen so that the easy ways to boost it are also good for users.

Canonical shapes

The "compound" trap

"DAU × sessions per DAU × actions per session" feels comprehensive but obscures which lever is moving. Pick one metric, accept the loss of granularity, and rely on a secondary set to triangulate.

Leading vs lagging indicators

Lagging indicators reflect the final business outcome — revenue, retention, NPS. They're authoritative but slow. Leading indicators predict the lagging outcome and move faster. Good teams instrument both and use leading indicators to make decisions within a quarter without waiting for lagging confirmation.

Patterns

• Activation events (completed setup, first publish, first API call) lead retention.
• Engagement frequency (sessions/week) leads churn.
• Quality signals (rating, abandonment) lead refund / cancellation.
• Support volume on a specific surface leads cancellation tied to that surface.

Funnels and conversion

A funnel is the canonical product diagnostic. Define every step that gates the next, measure conversion step-by-step, find the biggest drop. Two things to get right:

Step definition

Each step must be unambiguous and time-ordered. "Signed up" → "uploaded a video" → "published a video" is clean. "Engaged with the product" is not — define it.

Denominator discipline

Conversion at step N can be measured against: (a) the step N-1 base (step-conversion), or (b) the funnel-top base (overall conversion). They diverge and matter for different questions. Stakeholders often want overall; product teams often want step-conversion. Always say which one you're showing.

Retention shapes

Retention curves come in three families. Knowing which family you're in shapes everything downstream:

• Decaying to zero — like a leaky bucket. Most consumer apps. Optimization target: shift the curve up at every horizon.
• Decaying to a floor — a stable "habit" cohort survives. Like Spotify or Slack. Optimization target: raise the floor.
• "Smile curve" — retention dips then rises (re-engagement). Common in transactional products. Means there's a re-engagement loop worth investing in.

Cohort vs aggregate

Always look at cohort retention, not aggregate "active users this week / active users last week." The latter conflates cohort effects with growth and can hide deterioration when new users mask the loss of old ones.

Guardrails

Guardrails are metrics you'd halt an experiment or roll back a launch on, even if the primary moved positively. The classic set:

• Latency / performance — p50, p95, p99. A primary that lifts conversion 5% by adding 2 seconds to TTI is usually not a ship.
• Crash / error rate — non-negotiable.
• Customer support tickets — a leading indicator of user pain that doesn't always show up in retention quickly.
• Revenue per user — even if conversion lifts, watch ARPU.
• Adverse-event metrics specific to the product — for AI: refusal rate, hallucination rate; for marketplaces: dispute rate; for fintech: chargeback rate.

Defining a metric from scratch

Common loop prompt: "We're launching X. What's the metric?" The script:

1. What's the decision the metric is going to inform? Ship/no-ship a feature? Allocate marketing budget? Set product strategy?
2. What's the unit of analysis? User, session, account, request.
3. Pick the metric definition: numerator, denominator, time window, inclusion rule.
4. State what it doesn't measure: every metric blinds you to something. Name what.
5. Name the secondary metrics that triangulate: the leading and lagging pair.
6. Name the guardrails: what would make you stop, even if the primary is positive.

Diagnosing a metric drop

"Conversion dropped 5% last week — what do you do?" The protocol:

1. Is it real? Check the data pipeline first. Did event volume drop? Did the schema change? Did a new release suppress an event? Half of "metric drops" are instrumentation breakage.
2. Is it the same denominator? If the funnel-top grew (a marketing push pulled in a wider audience), conversion-rate of that wider audience might be lower without anything getting worse for the original audience.
3. Slice it: platform, country, browser, traffic source, plan, new vs returning. Find where the drop concentrates.
4. Correlate with releases: what shipped that week? Both your team's releases and dependencies (upstream API changes, marketing campaigns).
5. Form a hypothesis: based on slices + releases, what's the most likely cause?
6. Validate or kill the hypothesis: confirm with a focused query, or design an experiment to test.
7. Communicate: a one-paragraph "what we know, what we suspect, what we're doing next" update to stakeholders. Update daily.

Interview probes