Lin Jia & Kexin Fei - The A_b Testing Blind Spot_ Solving The Opt-in Paradox With Causal Inference
**Target Audience**
* **Roles:** Data Scientists, Machine Learning Scientists, Applied Scientists, Economists.
* **Experience Level:** Intermediate.
**Required Background Knowledge**
Familiarity with standard A/B testing, introductory linear regression, and basic causal inference concepts (e.g., confounders, selection bias). Basic knowledge of Python’s scientific stack (e.g., `scikit-learn`).
**Talk Type and Approach**
Simulation-driven and Practical. The session will blend a live-coded Python simulation with mathematical intuition (DAGs) and a real-world product case study.
**Audience Takeaways**
* **Conceptual:** Visually understand why standard observational methods (like Propensity Score Matching or naive Doubly Robust ML) structurally fail for opt-in features due to collider bias.
* **Strategic:** Learn how to disentangle a product's intrinsic value from its marketing reach, preventing the business from abandoning highly effective features simply due to poor adoption UX.
* **Practical:** Gain a Python blueprint for implementing Randomized Encouragement Design (RED) and the Interactive Instrumental Variable Model (`DoubleMLIIVM`) to handle heavy-tailed, highly skewed platform data.
**Technical Depth & Educational Value**
This talk moves beyond theory by using code-driven simulations to visualize causal failures, providing a robust structural solution for each part of the Trilemma:
* **Solving Voluntary Adoption (Simulation & RED):
* **Solving Heterogeneity
* **Solving Finite Samples:**
Minutes 0-5: The Business Context & The Trilemma.** Introduce the opt-in paradox across platform ecosystems. Define the "Trilemma" preventing us from measuring true ROI: Voluntary Adoption, Extreme Heterogeneity, and Finite Sample Sizes.
Minutes 5-15: The Measurement Trap & RED Framework.** Using a Python simulation, we visualize the failure of standard A/B tests (dilution) and observational models (Collider Bias). We solve this conceptual hurdle by introducing Randomized Encouragement Design (RED) to create a valid instrument.
Minutes 15-20: The Variance Problem & DoubleML.** We explain why standard IV estimation (`2SLS`) isn't enough for highly skewed user data. We walk through the `DoubleML` Python implementation (`DoubleMLIIVM`), highlighting the use of `Tweedie` Regressors to handle zero-inflated data and recover the precise LATE.
Minutes 20-25: Empirical Case Study.** We apply the full RED + `DoubleML` framework to a real-world product decision, demonstrating step-by-step how it recovered the signal from the noise and saved a high-potential feature from being abandoned due to poor UX discovery.
Minutes 25-30: Summary, Learnings & Q&A.** A quick recap of the architectural blueprint (the "how-to" for attendees). Open the floor for Q&A.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases.
00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Want to help add timestamps to our YouTube videos to help with discoverability? Find out more here: https://github.com/numfocus/YouTubeVideoTimestamps