Project:
A/B Testing Design Process
Enabling Experimentation: Establishing a Scalable A/B Testing Design Process
Micro-study
Background
Company:
Pizza Hut
Role:
Associate Manager, UX Design
Collaborative Partnerships:
Pizza Hut Data Analytics Team
Optimizely
Overview & Objectives:
This micro-study highlights the creation of a formalized design process to support a robust and scalable A/B testing pipeline. The initiative aimed to streamline the delivery of impactful design concepts while ensuring consistent and reliable measurement of their outcomes.
Scope of Responsibility:
I identified workflow challenges, established an A/B testing design process, and worked closely with cross-functional teams to align with program goals. From there, I created a Confluence space to track concepts, developed a reusable Figma template, onboarded UX team members to ensure smooth adoption, and piloted the new process across multiple active projects.
Tools:
Figma
Confluence
Jira
Optimizely
Process
Key Challenge:
The absence of a formal A/B testing process within the cross-functional team created inefficiencies in planning, execution, and analysis, hindering reliable impact measurement and rapid iteration. A structured, scalable workflow was needed to align the team, streamline collaboration, and support data-driven, impactful design solutions.
Process Design:
Shifting the Approach
From
No centralized tracking of test concepts
To
A Confluence page capturing testing concepts, tickets, and results for better organization and visibility
From
Lack of clear design prioritization and visibility
To
UX Jira tickets to give cross-functional teams clear visibility into design priorities for new concepts and progress on existing ones.
From
A single, disorganized Figma file containing all test designs
To
Dedicated Figma files for each concept, allowing for more iteration and scalability, and resulting in clearer, production-ready designs.
Revised Testing Process
Pilot Project
Sticky Localization Banner
Hypothesis: Introducing a sticky localization banner will improve conversion by prompting users to select between delivery and carryout earlier in their shopping journey, allowing them to see local pricing and begin adding items to cart.
Key Metrics:
Localization rate
Conversion Rate
Test Findings:
Control:
Localization Rate: 66.8%
Conversion Rate: 28.0%
Variant 1:
Localization Rate: +1.34%
Conversion Rate: +1.13%
WINNER!
Variant 2:
Localization Rate: +1.30%
Conversion Rate: +0.80%
Decide & Deploy:
Winning Variant Impact: Based on the localization and conversion rates of Variant 1, the estimated annual conversion lift is $11.2M. This design variant was delivered to the Merchandising Optimization team and deployed to prod in November 2024.