首页 > 译 文 > 【观点】为什么产品经理不应该被数据驱动
2022
11-22

【观点】为什么产品经理不应该被数据驱动

本文仅为翻译,不代表译者任何观点


Today’s world of product development couldn’t be more different. It’s really easy to measure everything in your product – user behavior, feature adoption, conversion rates. With this comes the ability to accurately track how changes to the product impact all of these metrics, not just anecdotally but with cold, hard data.

今天的产品开发世界已经完全不同了。衡量产品中的一切——用户行为、特征采用、转化率——真的很容易。有了这种能力,我们就可以准确地跟踪产品的变化如何影响所有这些指标,不是通过坊间传闻,而是使用冷冰冰的数据。

The ability to make product decisions based on data has a lot of appeal. Tastes and convictions can differ, and nothing settles an argument better than simply trying different approaches and measuring the results. And while product management is always both an art and a science, making product decisions based on data tilts the balance toward the “science” side, which is appealing to many STEM-educated people involved in product development.

基于数据做出产品决策的能力具有很大的吸引力。品味和信念可能不同,没有什么比尝试不同的方法并衡量结果更能解决争论了。虽然产品管理通常既是一门艺术也是一门科学,但基于数据做出产品决策会使天平向“科学”一侧倾斜,这对许多接受过STEM教育的产品开发人员很有吸引力。

The ultimate form of this is the “data-driven” product manager, who bases product decisions on data as much as possible, and will try to set up quantifiable criteria for all product development. Improvement opportunities are identified based on where the metrics lag behind benchmarks. Changes to the product are evaluated based on how much they were able to move key performance indicators (KPIs) that were defined up front.

这种情况的最终形式是“数据驱动型”产品经理,他们尽可能将产品决策建立在数据的基础上,并试图为所有产品开发建立可量化的标准。根据度量落后于基准的地方确定改进机会。对产品的更改是基于它们能够移动预先定义的关键性能指标(KPI)的程度进行评估的。

There are many good aspects of this approach. Instrumenting the product, collecting quantitative evidence, and setting up success criteria before launching product changes are all practices that I would highly recommend. However, going fully “data-driven” is a dead end, and here’s why.

这种方法有很多优点。在启动产品变更之前,对产品进行测试,收集量化证据,建立成功标准,这些都是我强烈推荐的做法。然而,完全“数据驱动”是一条死路,原因如下。

The perils of Goodhart's Law

古德哈特定律的危险

One of the fundamental flaws of the data-driven approach is that it relies on the unshaken belief in KPIs. After rigorous analysis of the data, you define good metrics to measure that reflect the critical drivers of your product’s customer and business value, and then you tune your product development engine to keep improving those KPIs. Sounds great, doesn’t it?

数据驱动方法的一个基本缺陷是,它依赖于对KPI的坚定信念。在对数据进行严格的分析之后,您可以定义良好的度量标准,以反映产品的客户和业务价值的关键驱动因素,然后调整产品开发引擎以不断改进这些KPI。听起来不错,不是吗?

“Not so fast” calls out Charles Goodhart, wielding his law like a hammer:

“别这么快”查尔斯·古德哈特喊道,挥舞着他的法律就像一把锤子:

When a measure becomes a target, it ceases to be a good measure.

当一个度量成为一个目标时,它就不再是一个好的度量。

What does this mean? It means that as soon as you make a measure a KPI and start trying to improve it, the measure is no longer useful. Let’s look at some examples. Let’s say your measure is daily active users, often used as a KPI because it is a proxy for how many users are getting value out of your product on a daily basis. Now, you send a bunch of push notifications, the number goes up. Did your product improve? Hardly. Did more people get value out of the product? Debatable.

这是什么意思?这意味着一旦您将一个度量作为KPI并开始尝试改进它,该度量就不再有用了。让我们看一些例子。假设你的衡量标准是日活跃用户,这通常被用作KPI,因为它代表了每天有多少用户从你的产品中获得价值。如果你发送一堆推送通知,数量就会上升。你的产品改进了吗?几乎没有。更多的人从产品中获得了价值吗?这是有争议的。

The issue here is that most measures are only proxies, leading indicators that are hopefully correlated to some lagging indicator that we are actually interested in (like retention, some monetization event, or customer lifetime value).

这里的问题是,大多数度量只是代理,领先指标与我们真正感兴趣的滞后指标(如留存率、某些盈利事件或用户终身价值)相关。

It gets worse than that, however. Even if we take measures that aren’t proxies, Goodhart’s Law still applies. Let’s use revenue – not a proxy, just actual money flowing to the company, and clearly a necessary prerequisite for a thriving business. However, I can tell you a surefire way to grow revenue: sell dollar bills for ninety cents. Of course, no company is doing precisely that, but there are many that do something similar enough: spend more to acquire customers than they are getting back in customer lifetime value. If you spend a dollar on Facebook ads to acquire a customer with a lifetime value of ninety cents, then that is exactly equivalent to the dollar bill example. “Sure, we’re losing ten cents on every sale but we’ll make it up in volume” is an age-old joke that doesn’t lose its relevance. (For prominent real-life examples, look at ride-hailing companies. With billions of VC money poured into growth, it’s still not clear whether there is a sustainable business model there.)

然而,情况比这更糟。即使我们采取的措施不是代理,古德哈特定律仍然适用。让我们使用收入——不是一个代理,而是流入公司的实际资金,这显然是业务繁荣的必要前提。然而,我可以告诉你一个肯定能增加收入的方法:以90美分出售1美元纸币。当然,没有一家公司会这么做,但有很多公司做了类似的事情:花更多的钱来获取客户,而不是从客户终身价值中获得回报。如果你在Facebook广告上花1美元去获取一个终身价值为90美分的用户,那么这便等同于1美元纸币的例子。“当然,我们每卖出一笔就会损失10美分,但我们会通过数量来弥补”是一个古老的笑话,但它并没有失去相关性。(对于现实生活中突出的例子,可以看看网约车公司。随着数十亿风投资金涌入,是否存在可持续的商业模式仍不清楚。)

Let’s make profits the KPI, then. Surely that must be it. Profits don’t lie. If I increase profits, then I must be doing something right? Sadly, no. Firing your whole engineering team will certainly increase profits in the short term… but does anyone believe that you can build a sustainable business that way?

让我们将利润作为KPI。肯定是这样。利润不会说谎。如果我增加了利润,那么我一定做了什么,对吧?很遗憾,没有。解雇整个工程团队肯定会在短期内增加利润,但有人相信你可以通过这种方式建立一个可持续的企业吗?

In the end, taking into account Goodhart’s Law, you need to always be careful and qualitatively reason whether the changes you are observing are real, sustainable changes or whether the connection between the KPI and the long-term success driver of the business has been lost.

最后,考虑到古德哈特定律,您需要始终小心谨慎,并从定性上推断您所观察到的变化是真实的、可持续的变化,还是KPI和业务的长期成功驱动因素之间的联系已经丢失。

备注:

古德哈特定律:这个定律可以有多种解释,其中最常见的一种解释是:一项社会指标或经济指标,一旦成为一个用以指引宏观政策制定的既定目标,那么该指标就会丧失其原本具有的信息价值。因为政策制定者会牺牲其他方面来强化这个指标,从而使这个指标不再具有指示整体情况的作用。

换一种简洁的说法是:一项指标一旦成为政策制定的依据,便立刻不再有效。政策制定者会牺牲其他方面来强化这个指标,使得这个指标不再具有指示整体情况的作用

---来源于百度百科

Baby steps to climb a hill

婴儿爬山

Another issue with the data-driven approach is that it favors incremental improvements. In general, it is far easier to find small tweaks that have a small positive impact on the KPI than it is to find big changes with big positive impacts. The reason for that is twofold: firstly, you can test small changes much more quickly. Google’s infamous “42 shades of blue” test is a good example. Hardly any effort required, just test different colors until you find one that slightly improves the conversion rate. Success! Donuts all around, we shipped a winning feature.

数据驱动方法的另一个问题是它倾向于增量改进。一般来说,找到对KPI有小积极影响的小调整要比找到有大积极影响的大更改容易得多。这样做的原因有两个:首先,您可以更快地测试小更改。谷歌臭名昭著的“42 shades of blue”测试就是一个很好的例子。几乎不需要任何努力,只要测试不同的颜色,直到你找到一个稍微提高转化率的颜色。成功!甜甜圈到处都是,我们发布了一个获胜的特征。

Secondly, these small incremental wins often lead to the current solution approaching a local maximum. You have optimized the heck out of the current solution, it runs like a well-oiled machine. If you now test a different solution, a more radical change, against it, it won’t be as optimized. If you only analyze how it performs, as measured by cold, hard data, it often won’t beat the optimized control experience – but that might just be because it’s not yet living up to its potential.

其次,这些小的增量胜利通常会导致当前解决方案接近局部最大值。你已经优化了当前的解决方案,它就像一台运转良好的机器。如果你现在测试一个不同的解决方案,一个更激进的改变,它不会得到优化。如果你只分析它的表现,就像用冷冰冰的硬数据来衡量,它通常不会超过优化的控制体验——但这可能只是因为它还没有发挥出它的潜力。

You can try teaching your product managers and product teams that failure is part of the job, that they should shoot for the moon and perhaps land among the stars. The truth is, though, that everyone likes to be successful. It’s natural to prefer running five small test out of which one or two were winners to running one big test which failed.

你可以试着教导你的产品经理和产品团队,失败是工作的一部分,他们应该向月球进发,也许还可以在星星之间着陆。然而,事实是,每个人都喜欢成功。比起运行一个失败的大型测试,我们更倾向于运行五个小型测试,其中有一两个是赢家。

Moreover, because of the local maximum problem, if you are strictly data driven, you never know if you landed on a lower hill or just on the foot of a bigger hill. The data won’t tell you that. Which leads directly to the next point…

此外,由于局部极大值问题,如果你完全受数据驱动,你永远不知道你降落在的是较低的山脚下还是较大的山脚下。数据不会告诉你这些。这就直接引出了下一点……

Solving the hard problems

解决难题

The best products weren’t built by starting with a KPI and then trying to find ways to improve it. The best products were built by focusing on a problem and banging your head against the wall trying to solve it. Being data-driven can lead to prioritizing thinking opportunistically rather than strategically.

最好的产品不是从KPI开始,然后试图找到改进它的方法。最好的产品是通过专注于一个问题,并用头撞墙的方法来解决它。数据驱动会导致机会主义思维而非战略思维。

There are of course good reasons to act opportunistically. If you might unlock a big new sale or partnership by doing something that is ever so slightly adjacent to what you are doing currently, it might be wise to leap on that opportunity. However, the big question here should be whether that furthers the strategic goals of the company, not whether it drives up some KPI or another. Otherwise, it becomes a goose chase – an unfocused effort on doing whatever it takes to move KPIs, rather than being really, really good at solving the core problem that the product should be solving.

当然,采取机会主义行动是有充分理由的。如果你可能通过做一些与你目前所做的稍微接近的事情来获得一笔新的大销售或合作,那么抓住这个机会可能是明智的。然而,这里的大问题应该是这是否促进了公司的战略目标,而不是它是否推动了一些KPI或另一些。否则,它就变成了徒劳的追逐——没有集中精力去做任何移动KPI所需的事情,而不是真正非常善于解决产品应该解决的核心问题。

Perfect is the enemy of done

完美是完成的敌人

The last major issue with being data driven is that it can lead to slowing down the organization. If decisions aren't made until sufficient data is available, then decisions will be made later than they should be, on average. This is a bit counter-intuitive at first – after all, product analytics data is usually available in real time, whereas gathering more qualitative feedback from customers requires time to collect.

数据驱动的最后一个主要问题是,它可能导致减慢组织的速度。如果在有足够的数据之前不做决定,那么平均来说,决定将比应该做的更晚。乍一看,这有点违背直觉——毕竟,产品分析数据通常是实时的,而从客户那里收集更多的定性反馈则需要时间。

However, you can collect qualitative feedback much earlier in the development lifecycle than you can measure impact quantitatively. You can put a low fidelity prototype in front of a handful users before a line of code has been written. An A/B test, on the other hand, requires a reasonably functional implementation.

然而,在开发生命周期中收集定性反馈的时间要比定量度量影响的时间早得多。在编写一行代码之前,你可以将一个低保真度的原型放在少数用户面前。另一方面,A/B测试需要一个合理的功能实现。

Data-driven product managers will scoff at the sample size of just doing a few prototype testing sessions. However, a lot of times even after five to ten interviews, you will see the findings converge. Moreover, often it will even take only a single conversation with a customer to uncover a flawed assumption underlying the current approach.

数据驱动型产品经理会嘲笑只进行几次原型测试的样本量。然而,很多时候,即使是在五到十次访谈之后,你也会看到结果趋于一致。此外,通常只需要与客户进行一次对话,就可以发现当前方法背后的错误假设。

Do A/B testing and other quantitative validation approaches yield more “accurate”, reliable results? Perhaps. But in a world in which speed to market and pace of continuous innovation often determine who emerges the winner, and where the next hungry startup is always waiting to disrupt any incumbent who gets too complacent, you don’t have the luxury of waiting for perfect information.

A/B测试和其他定量验证方法能产生更“准确”、可靠的结果吗?也许。但在这个世界上,进入市场的速度和持续创新的速度往往决定了谁是赢家,下一个饥渴的初创公司总是等着颠覆任何过于自满的现任者,你没有等待完美信息的奢侈。

The solution: Be data-informed, not data-driven

解决方案:了解数据,而不是受数据驱动

You shouldn’t take away from this that relying on data to make product decisions is inherently bad. In fact, it can be really useful, as stated at the top of this article. Accurate knowledge of how your product is being used is really valuable, so don’t disregard that. However, don’t let the data drive you. Become data-informed, instead. Instrument your product, use that data to inform areas of the product that don’t work as well as others. Use A/B testing if you are trying to optimize something, by all means – if you are trying to find the best-performing copy on your paywall, an A/B test is often the fastest and most reliable way to get results. Also, absolutely do define success measures for product changes and go back after the fact to validate that the changes did what you hypothesized.

你不应该否认依赖数据来做产品决策本身就是不好的。事实上,正如本文开头所述,它可能非常有用。准确了解你的产品是如何被使用的是非常有价值的,所以不要忽视这一点。然而,不要让数据驱动你。相反,要成为数据通。测量你的产品,用这些数据来告诉你产品的哪些方面不如其他方面。如果你想优化某些内容,请务必使用A/B测试——如果你想在付费墙上找到性能最好的副本,A/B测试通常是获得结果的最快、最可靠的方法。同样,一定要定义产品变更的成功度量,并在事实发生后回去验证变更是否达到了您的假设。

On the other hand, also balance that quantitative perspective with a qualitative one. Ground your product decisions in mission and strategy, not just in how to achieve quantitative goals. Talk to customers. Test low-fidelity prototypes. And above all, question whether data and intuition (“product sense”) tell you the same thing. If something should move metrics and it doesn’t, or it shouldn’t move metrics and it does, ask yourself “why”. Better yet, don’t just ask yourself, but use user research to find out why. The answers to these questions will tell you more about your customers and your product than only looking at numbers ever would.

另一方面,也要用定性的观点来平衡定量的观点。把你的产品决策建立在使命和战略的基础上,而不仅仅是如何实现量化目标。与顾客交谈。测试低保真度原型。最重要的是,质疑数据和直觉(“产品感”)是否告诉你同样的事情。如果某些内容应该改变参数却没有改变,或者不应该改变参数却改变了,那就问问自己“为什么”。更好的做法是,不要只问你自己,而是通过用户研究来找出原因。这些问题的答案会让你比只看数字更了解你的客户和你的产品。


本文》有 0 条评论

留下一个回复