⦾ The Project

A novelty subscription box service was looking to improve their first-time customer flow experience which I expanded into an experiment to generate learnings about customer growth. 

Platform ‣ Hygge Box is a Scandinavian novelty box service that promotes a lifestyle of comfort and simplicity via monthly shipments of candles, hot cocoa, blankets, etc.
Brief ‣ Improve the first-time customer flow in any way you see fit 

Research ‣ Secondary research on existing materials; experiment design, scenario modeling the proposed experimentation
Suggested team ‣ Product designer, engineer with A/B test experience, product manager good at data analysis, marketing rep with good sales 
Working methodology ‣ Agile sprints

⦾ The Objective 

Goal ‣ Using existing public data, acquire new users from the current target segment

Key metric ‣ Increase in % of first-time users who purchase at least 3 months worth of boxes

Why choose this key metric? ‣ Based on preliminary research that suggested converted customers quickly feel a deep emotional connection to the product, I hypothesized that the (currently known) highest value users are ones who fall in love immediately and invest in more than just 1 single trial box

⦾ The Work

Define the target user & key metric ‣ Induce the value to the end user by fielding product reviews and cultural competitors to get a sense for who a loyal, first-time customer is and what they might find valuable about Hygge Box

Establish research goal  

• Heuristic evaluation of current experience to identify which parts of the website already are strong in simplicity and warmth
  
• Compare those areas with social media-induced value-to-user to come up with a key growth hypothesis
• Using this hypothesis to drive the experiment, we will learn about the usability of our home page, our onboarding’s effect on new user acquisition, and the value that drives new users to sign up and purchase

Create research plan 

• 3-4 sprints of A/B tests, with A as control (current website) and B as new 3-step process, iterating each week based on results from prior week
• Work with organic web traffic for this experiment 
• Track # of new users, categorize by # of months/boxes on first purchase 
• Use eye-tracking and click analytics to make inferences about user behavior week-to-week

Create research apparatus (high-fi homepage mockups showcasing a streamlined, 3-step process)

Conduct research

• Push A/B versions live each week 
• Review data as it comes in, and use behavioral inferences to inform iterations on design B from week to week 
• Iterate B for sprint 2 
• Iterate B for sprint 3 
• Iterate B for sprint 4

⦾ End result

Interpret results to distill learnings ‣ Work with the PM to answer the following:

• Did we meet the KPI? Did the % of new users who buy at least 3-months worth of boxes increase?
  
• If rate of acquisition grew but we did not meet KPI, did we choose the wrong key metric?
• Looking at the differences in data from week-to-week, what can we infer about the effectiveness of the various iterations? 

⦾ Reflection

Next steps  

• If we failed in growing ‣ Conduct discovery interviews with 5-10 new users who signed up during the experiment to understand why they did not purchase more — perhaps we are targeting the wrong segment, or perhaps we need a different approach 
  
  
• If we succeeded in growing, but not according to our metric ‣ Repeat the experiment, but instead of live A/B tests, run unmoderated interviews with a self-directed questionnaire using recruited users who each try out one of the 4 iterations of design B (the new design) — this would help us discover what is actually causing these users to purchase more than other new users and therefore determine a more impactful key metric 
  
  
• If we succeeded in growing according to our metric ‣ I would conduct just ~5 discovery interviews with the actual new users who signed up during this experiment, to probe deeper into their behavior and help us better understand our ideal high-value user