A/B Testing at Scale - Risks and Opportunities

Booking.com is known for running A/B tests at scale to optimise every small and big change to our products. Guiding product development by monitoring well-defined metrics allows dozens of teams to work on the same product simultaneously and iterate quickly. However, A/B testing methods can easily be misunderstood, unintentionally abused, or become a limiting factor. 

In this talk, we present common pitfalls and shortcomings of classical A/B testing we've experienced at Booking, together with some concrete remedies and alternative solutions. After listening to this talk, you can go back to your teams and constructively challenge them.