Observations on Data, Metrics and Goals

Again, this is not an exhaustive survey, but a collection of observations of things I’ve found often useful or often overlooked.

1. If your team has seven goals, you probably missed some. Max three — you can’t handle the tradeoff complexity beyond that.
2. When a team is investing for the long term, they’re really saying it’s negative in the short term. Else they’d be claiming the win already.
3. If a metric looks slightly out of line, it’s slightly out of line. If it looks way out of line, your method of evaluation is probably broken.
4. Garbage in, garbage out. If in doubt, it’s probably a bug in your ETL pipeline. They’re usually a nightmare to debug without a solid ground truth.
5. How many N’s do you need to tell if a medicine works? Only one if the first patient dies. Don’t run experiments at scale if you know already know they don’t work.
6. Data scientists get job security by being the only people who know which metrics to trust. But eventually everyone gets slowed down if all knowledge has to flow through them.
7. Everyone (PM, Eng, DS, Des) should be able to say what happened from the data. PM, DS should be able to say why it happened from the data. DS really comes into it’s own predicting what will happen next.
8. Don’t waste data scientists’ time asking them to continually explain what or why. Nothing annoys them more. Either dig in yourself, or invest in the tooling to make it easy for everyone to do so.
9. Know the confidence intervals around a metric before you send people off to explain why it’s up/down this week. Explaining noise is such a giant waste of time.
10. Explain variance to forecast, not variance to hope. Start by asking what of your plan didn’t happen, for which you need a plan to begin with. If you didn’t know what to expect would happen, its hard to explain what did.
11. If a result is positive, show net. If it’s negative, show gross.
12. Goals are three things: a metric, a value, a moment in time. When a goal feels wrong its usually missing one of these, or they’re not specific enough.
13. Always set a goal, even if only to force you to be thorough about the levers, relationships and tradeoffs within your product.
14. Push the value of a goal until it would require you to completely change your roadmap. What would you do differently if you wanted a 2X, 5X, 10X different outcome?