Shimmer: 0→1 AR iOS Retail App

Roles: Designer｜Read time: 5 mins

      

I joined FlyBy Media, an early-stage computer vision startup as their second design hire to prove our patent-pending image recognition technology through a consumer AR shopping app. I designed 5 core user flows, net-new AR interaction patterns, and in-store navigation features that would validate the tech's real-world reliability. These user experiences ultimately contributed to Apple's acquiring us for our computer vision technology for their first self-driving cars.

MVP Core Flow: Unlocking Brand Stories by Scanning Products In-Store

After tapping a product in the feed, the user is either shown directions to the store, or if beacon tech determines they are in-store, they are met with an in-store map that helps them find the product. They can then "unlock" the item, allowing them to consume the media/story behind the product.

Key Feature: In-Store Navigation & In-App Apple Maps

I partnered with engineers to define requirements for an in-store mapping system that used beacon technology to detect when users entered partner stores. Using Apple Maps as the design basis promoted ease-of-use among our iPhone users—the experience seamlessly integrated with embedded Apple Maps to help you find the store, then switched to our custom map once inside.

Our reps would scan the inside of partnering stores, and the system would load using map elements from the design system I developed. This spatial design work proved our computer vision could handle complex, real-world retail environments beyond simple product recognition.

Post-Launch Iteration: In-App Rewards for Product Discounts

After launch, we received feedback that indicated users felt a lack of value in the core unlock-flow. I partnered with our sales and marketing team and brainstormed ideas. I proposed that the app provide QR code-based discounts as a reward for using the app to unlock stories. We decided to move forward with that idea. 

Tech Validation & Strategic Impact

Rather than traditional e-commerce conversion metrics, success was measured by engagement data that proved our computer vision technology worked reliably in real-world conditions. High session lengths and product recognition accuracy rates demonstrated the tech was ready for larger applications.

The validation worked—Apple acquired our computer vision technology and integrated it into their self-driving car program. Our consumer app had successfully proven that the underlying technology could handle the complex visual recognition challenges needed for self-driving cars.