Mastering Recommendation Systems Evaluation: An A/B Testing Approach with Insights from the Industry

Recommendation systems shape personalized experiences across various sectors, but evaluating their effectiveness remains a significant challenge. Drawing on experiences from industry leaders such as Booking.com, this talk introduces a robust, practical approach to A/B testing for assessing the quality of recommendation systems. The talk is designed for data scientists, statisticians, and business professionals, offering real-world insights and industry tricks on setting up A/B tests, interpreting results, and circumventing common pitfalls. While basic familiarity with recommendation systems and A/B testing is beneficial, it's not a prerequisite.

This talk aims to provide attendees with a practical understanding of A/B testing in the evaluation of recommendation systems, including unique insights from industry practices and specific tricks that enhance effectiveness.

My report includes next steps:
- Introduction to recommendation systems, their ubiquity, and the imperative for evaluation, including industry examples.
- An overview of A/B testing and its vital role in assessing recommendation systems, supported by insights from Booking.com and other industry leaders.
- Techniques for designing effective hypotheses for A/B tests, focusing on recommendation systems.
- Choosing pertinent metrics for robust evaluation of recommendation systems with industry examples.
- Conducting A/B tests: industry best practices, common pitfalls, and strategies for mitigation, reinforced by real-world cases.
- Accurate interpretation of A/B testing results and management of statistical biases, with insights from the field.
By the end of the talk, attendees will have a comprehensive understanding of how to apply A/B testing effectively to recommendation systems, select relevant metrics, interpret results accurately, and navigate common challenges, backed by industry best practices and practical examples.

Bio:
Ildar Safilo
Machine Learning Scientist in the Booking.com
Experienced manager in MLE/DS/SE/DA, I possess extensive expertise in machine learning, analytics, and software engineering. I excel at leading teams to create groundbreaking businesses and delivering innovative solutions for real-world business cases across various industries, including IT, banking, telecommunications, marketplaces, game development, shops, Travel-tech and streaming platforms.
Expert in building recommendation and ranking systems, as well as personalization automation with machine learning, and advanced A/B testing.
Co-author and lecturer of a popular online course on recommender system development with over 1000 students.
Co-author an open-source Python library called RecTools, specifically designed for building recommender systems. The library is hosted on GitHub at RecTools and has received widespread recognition and adoption in the industry.
Graduate with a Master’s degree in Mathematics and Computer Science and over 6 years of experience in data science.

===

www.pydata.org

PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.

PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.

00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.

Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVi...