Ambient Experience & Generative AI

Bring art, generative AI and widgets to the Fire TV for the first time.

My Role

Lead Designer

Company

Amazon

Platform

FireOS & APL

Scope

New Feature Development

Art on the biggest screen in the home

With the launch of the 2, 4, and Omni series TVs, Amazon expanded its product portfolio beyond streaming devices to incorporate fully integrated televisions. This shift opened up new opportunities for enhancing customer experiences, given that Amazon now controls both the software and hardware aspects of the TV.

Recognizing that a significant portion of Fire TV usage occurs after 7 PM, we identified an opportunity to transform the TV—a large and often underutilized display in homes—into a more engaging focal point throughout the day. This led to the development of Ambient Experience, which aims to convert traditional TVs into stunning 4K art displays when not actively in use. Rather than remaining a dormant black rectangle, the TV becomes a canvas showcasing over 1,000 pieces of artwork at launch, featuring everything from classic impressionist paintings by Monet and Cézanne to breathtaking landscape photography from acclaimed artists worldwide.

The design process for Ambient Experience was different from any previous projects. A fundamental design principle was “Minimize Distraction,” which emphasizes creating a user interface that seamlessly fades into the background, allowing the artwork to take center stage.

I led a team of three designers—two interaction designers and one specializing in visual design—to outline the various segments of the experience, including Core (art and widgets), Art Gallery, Widget Gallery, and Preferences. My primary responsibility was overseeing the Core CX while guiding the other UX designers in their respective areas.

Generative AI

Text-to-image AI models were mostly found in chat-based LLMs by late 2023. Since art is essential to the Ambient Experience, we aimed to let customers make art just using their voice. The project began as a three-week effort to create a proof of concept with existing tools. After giving developers a basic interface, I focused on exploring and learning about image generation AI models. The team created a prototype using open-source Stable Diffusion v1.5 model, which needed significant prompt engineering for good results. The techniques I learned helped us develop a prompt recommendation engine that discreetly enhances customer prompts. We also tried LoRA/Dreambooth for more creative options.

I proposed a feature to tackle the "blank canvas" issue noticed after initial launch. Many customers struggled to know what to ask since the feature's capabilities weren't clear. (The common hesitation with using voice assistants added to this problem.) However, the strength of the experience comes from the limitless possibilities of open prompts, so I didn't want to limit choices with a GUI. I decided on a "Surprise Me!" widget that offers customers a new prompt daily from a curated library and customer can see the widget when Ambient Experience comes up on their TV. Customers can click on the prompt to see the generated images or browse the library for more options. We invested time in creating a large library to give customers a sense of endless choices and hopefully inspire them to create their own prompts.

Widgets

Widgets provide concise information that can be quickly accessed. Collaborating closely with a partner design system team, we developed a cohesive widget experience across Amazon devices, focusing on scalability and uniformity.

Customers familiar with widgets on Echo Shows can now enjoy the same experience on Fire TV devices, offered in two sizes. This integrated approach helps customers avoid the need to relearn widget functionalities, ensuring a seamless transition between devices.

My primary responsibility involved working with various domain teams to ensure that their widgets are optimized for TV. Additionally, I took the lead on a work stream aimed at adding remote interactions to a widget framework originally designed for touchscreens.