Concept

AB Testing

A method of comparing two versions of a product or feature by exposing different users to each and measuring the difference in outcomes to infer a causal effect.

Preparing graph…

Connection labelsColor by connections

100%

Concept

AB Testing

8 visible of 22 connections

18 nodes · 18 connectionsBookConceptEpisodeFrameworkPersonLarger = more connections+ Expand · − Collapse (double-click)Drag a node to pull its neighbors along

Related Concepts3

Related Frameworks24

Related Practices89

Ask about this concept

Transcript mentions

Jason Cohen 01:34:43
AB testing doesn't work very well and it doesn't work on most things. It won't work on strategy or vision or insights like nothing actually important to the suc...
Albert Cheng 00:56:27
So I just encourage taking that first step, just run an A/B test, find a third-party tool or something that you can integrate quickly or even just work with your engineers to spin something up. Just get in the practice of crawl then walk…
Julie Zhuo 00:34:07
And if you do a customer interview, that's still all data, it's just that that is a little harder to distill and quantify. Although now with AI, we have better tools for synthesizing.
Elena Verna 00:26:20
… We don't want to do user interviews, but we don't want to do A/B testing. It works for them.
Jackson Shuttleworth 00:00:30
'd say test everything, we've run in the last four years over 600 experiments on the streaks, so every other day. We've actually set up really good infrastructure for copy testing.
Brian Tolkin 00:40:29
Yeah, very hot topic of conversation. We do A/B test.
Failure 00:49:51
And then they ended up asking me to come on full time, do that across the world, Japan, China, Korea, Turkey, Spain, France, et cetera. And then to own growth, which ended up meaning communications, social media, government partnerships,…
Failure 00:49:51
… cetera. And then to own growth, which ended up meaning communications, social media, government partnerships, anything to grow.
Ramesh Johari 00:42:56
So the big lesson of this Microsoft paper, it's called A/B Testing with what's called Fat Tails, which in lay terms just means you're running a business where there's potentially big opportunities out there if you look at the effects of…
Gina Gotthilf 00:45:12
… So first of all, I would actually say that AB testing and product-led growth is organic growth, because you're not spending money on it. And so the first thing that you can do, no matter what stage your ...
Itamar Gilad 00:54:03
So AB tests, multivariate tests, those are all experiments.
Tim Holley 00:17:40
… We've long at Etsy been a really, and maybe rose-tinted glasses speaking a little bit, but a really great experiment, A/B testing driven culture. And so when you think about things like retention, you can absolutely test, course I'm not…
Tim Holley 00:36:14
So like I said, I mentioned earlier, we've long been a very A/B testing driven organization, not least because Etsy's background and history has deep, deep roots in an incredible engineering culture. And so that's really tried and true.
Tim Holley 00:36:46
But the through line is, is the change that we're making adding value? And that's what we want to try to understand.
Ronny Kohavi 00:24:59
First of all, there are some necessary ingredients to A/B testing. And I'll just say outright, not every domain is amenable to A/B testing.
Ronny Kohavi 00:25:14
So you do have to have some necessary ingredient. You need to have enough units, mostly users, in order for the statistics to work out.
Ronny Kohavi 00:28:14
The OEC or the overall evaluation criterion is something that I think many people that start to dabble in A/B testing miss. And the question is, what are you optimizing for?
Gustav Söderström 00:56:45
… paradigms, et cetera, you're going to get replaced. You have to find this balance of trying new things.
Laura Schaffer 00:38:51
… And AB testing is one of the most expensive kinds of ways to validate an experiment. It often requires design and engineering and the PM or growth person or marketi...
Sanchan Saxena00:29:13
… There was no A-B testing that if you add stories at the top, I mean, there was a little bit of A-B testing, but not like, okay, let's go measure....
Gibson Biddle00:48:26
… That's the A-B testing. There's the management.
Julie ZhuoSource summary
Use experiments to test product assumptions.

💡

Concept

AB Testing

A method of comparing two versions of a product or feature by exposing different users to each and measuring the difference in outcomes to infer a causal effect.

Related Concepts3

Related Frameworks24

Related Practices89

Ask about this concept

Transcript mentions

Jason Cohen 01:34:43
AB testing doesn't work very well and it doesn't work on most things. It won't work on strategy or vision or insights like nothing actually important to the suc...
Albert Cheng 00:56:27
So I just encourage taking that first step, just run an A/B test, find a third-party tool or something that you can integrate quickly or even just work with your engineers to spin something up. Just get in the practice of crawl then walk…
Julie Zhuo 00:34:07
And if you do a customer interview, that's still all data, it's just that that is a little harder to distill and quantify. Although now with AI, we have better tools for synthesizing.
Elena Verna 00:26:20
… We don't want to do user interviews, but we don't want to do A/B testing. It works for them.
Jackson Shuttleworth 00:00:30
'd say test everything, we've run in the last four years over 600 experiments on the streaks, so every other day. We've actually set up really good infrastructure for copy testing.
Brian Tolkin 00:40:29
Yeah, very hot topic of conversation. We do A/B test.
Failure 00:49:51
And then they ended up asking me to come on full time, do that across the world, Japan, China, Korea, Turkey, Spain, France, et cetera. And then to own growth, which ended up meaning communications, social media, government partnerships,…
Failure 00:49:51
… cetera. And then to own growth, which ended up meaning communications, social media, government partnerships, anything to grow.
Ramesh Johari 00:42:56
So the big lesson of this Microsoft paper, it's called A/B Testing with what's called Fat Tails, which in lay terms just means you're running a business where there's potentially big opportunities out there if you look at the effects of…
Gina Gotthilf 00:45:12
… So first of all, I would actually say that AB testing and product-led growth is organic growth, because you're not spending money on it. And so the first thing that you can do, no matter what stage your ...
Itamar Gilad 00:54:03
So AB tests, multivariate tests, those are all experiments.
Tim Holley 00:17:40
… We've long at Etsy been a really, and maybe rose-tinted glasses speaking a little bit, but a really great experiment, A/B testing driven culture. And so when you think about things like retention, you can absolutely test, course I'm not…
Tim Holley 00:36:14
So like I said, I mentioned earlier, we've long been a very A/B testing driven organization, not least because Etsy's background and history has deep, deep roots in an incredible engineering culture. And so that's really tried and true.
Tim Holley 00:36:46
But the through line is, is the change that we're making adding value? And that's what we want to try to understand.
Ronny Kohavi 00:24:59
First of all, there are some necessary ingredients to A/B testing. And I'll just say outright, not every domain is amenable to A/B testing.
Ronny Kohavi 00:25:14
So you do have to have some necessary ingredient. You need to have enough units, mostly users, in order for the statistics to work out.
Ronny Kohavi 00:28:14
The OEC or the overall evaluation criterion is something that I think many people that start to dabble in A/B testing miss. And the question is, what are you optimizing for?
Gustav Söderström 00:56:45
… paradigms, et cetera, you're going to get replaced. You have to find this balance of trying new things.
Laura Schaffer 00:38:51
… And AB testing is one of the most expensive kinds of ways to validate an experiment. It often requires design and engineering and the PM or growth person or marketi...
Sanchan Saxena00:29:13
… There was no A-B testing that if you add stories at the top, I mean, there was a little bit of A-B testing, but not like, okay, let's go measure....
Gibson Biddle00:48:26
… That's the A-B testing. There's the management.
Julie ZhuoSource summary
Use experiments to test product assumptions.