Veteran US Democratic congressman Elijah Cummings, who was heavily involved with the Trump impeachment inquiry, has died at the age of 68. His office said he had died as a result of "longstanding health challenges". As chairman of the House of Representatives Oversight Committee, he had […]
Business experiments, especially in digital formats like A/B testing, have exploded in the last decade. And for good reason. Experimentation promises the power of the scientific method to reduce uncertainty: Should we launch this product? Which messaging maximizes consumer engagement? Will this tool yield sufficient ROI when rolled out to all employees?
However, that promise comes at a price that few business leaders are prepared to pay. And as experiment evangelists, we’re partly to blame. In our enthusiasm, we don’t spend nearly enough time spelling out the organizational investments necessary to harness the full potential of this tool.
We hope it helps you prepare yourself to better invest in and implement experimentation.
Make sure you can measure. Experiments depend on measurement. If you can’t properly measure attribution from a digital ad to a sale, for example, you’ll have no luck running an experiment to figure out which ads are actually effective. Haven’t invested in good measurement yet? Do not proceed to #2.
Pay for a good translator. Too often, experiments are left to digital marketers or product managers that lack the statistical fluency to properly design, implement, and analyze experiments.
When hiring, be sure to evaluate the ability to communicate these concepts to product managers, marketers, and other collaborators.
ind a sandbox to play in. Before running experiments on anything with high stakes, try designing and running a simple A/B test from scratch in an environment that you entirely control. For instance, send a survey out to colleagues: invite half with one email and half with another, and see which version yields more opens and click-throughs. Figure out the power analysis by hand, thinking through the implications of each input, even if you have to Google every term or go to your in-house statistical expert for advice.
Spread your experimental eggs across several baskets. Business experimenting is like venture capitalism, not day trading. Big “wins” may be few and far between, but those winners will typically have outsized impact. As you move into more meaningful business experiments, assemble them into and launch them as “portfolios.” Run several treatments at once, if your sample affords it. If not, plan several experiments across different channels, to be run simultaneously or sequentially, but all under the same strategic umbrella.
Embrace the buddy system. As a part of spreading your bets, commit to experimenting with someone else in a different part of the organization. You will learn more from each other’s slip-ups and successes than you will from any textbook.
Make it public. Scientists across disciplines increasingly “pre-register” experiments, posting detailed designs and planned analyses publicly in advance of launch. (Medicine has done so for decades.) The practice helps catch errors, share learnings, and tie experimenters to the proverbial mast when they might be tempted to tweak results after the fact.
More than money, budget time. In spring 2018, Pandora published the results of an experiment addressing a fundamental question about their business: what level of ads pushes free users to subscribe, rather than leave the service altogether? The experiment took 21 months to complete and required a sample size of 35 million users. Experimental insights, even in relatively easier testing environments like digital products, take time and scale.
Most important, overhaul incentives. Like any new initiative, experiments often fail because of cultural “organ rejection.” They require taking short-term risks and often failing, all in service of long-term learning, and few businesses pat you on the back for failure even if you’re effectively taking one for the team.
Source: Harvard Business Review