Magic Leap | LuminOS Mixed Reality Platform
Mixed Reality UX + Visual Design
I spent nearly 18 months working with Magic Leap in the foundational stages of their new OS, LuminOS.
Project Goal
Create an experience robust and delightful enough to replace smartphones in people’s lives.
Magic Leap attempted to develop an entirely new mobile device category from the ground up. This required an operating system that lived in the real world and responded to the user’s actual environment. Nothing like this had ever been done at the time.
Platform apps and core features would all be developed simultaneously. The desired UX needed to be defined to ensure all development teams met the same standards for cohesion, intuitiveness, and delight.
I helped define the fundamental physics, metaphors, and interactions of the experience…
…that had to consider multiple input modalities, contexts, actions, and system requirements. I also led out in character, environment, and prop design as well as storyboards, color keys, and motion prototypes for VFX and interaction mechanics to define system-wide guidelines for many of the platform’s apps and core features.
The first six months were spent simply exploring how this OS would live in the user’s physical world. This led to a set of recommendations to guide the rest of the system design. I worked on Helio, which, at the time, was to be the world’s first MR browser, as well as Avatar Chat, or Social. I also spent a significant amount of time designing the out-of-the-box-experience (OOBE), which, as of this writing, is still in development.
My Roles
Ideation
Scenario Design
Framework/Metaphor Design
User Experience Design
UI Design
Concept Illustration
Presentation Design
The situation
How might we interact with a new platform that doesn’t exist?
How might we guide developers to make features and apps that feel coherently “Magic Leap?”
To deal with a project of this ambiguity, we began by listing and grouping all of the interactions we would need to account for in v1.
Selection
Activate
Modify
Set Value in Range
Single Select
Multi-Select
Modify > Continuous
Scroll
6DOF Manipulation
Constrained Manipulation
Make it Bigger
Annotation
Text Input
Modify > Discrete
Open
Close
Back / Home
Undo
Delete
Change Mode
Group
Ungroup
Share Locally
Share Remote
Combine
Map Environment
Pin to World
Design Principles
We then created design principles to guide the various teams working separately and concurently.
Authentic to the Medium
How does the photonic world look and feel?
Keep Us In the Moment
Mindful of the world and each other
Plays well with muggles
Comfortable and Efficient
100% reliable; makes me feel like a natural; Zero perceived lag
Safe to Explore
Give me a safe way back
Interaction Metaphors
We then brainstormed a ton of different interaction metaphor concepts that we could use to guide the UX/UI design. Concepts included craziness such as
Hermoine’s Bag
The Construct
Augment Me
Constellations
Magic
The Force
Swiss-Army Knife
Portals
Holodeck
Tangible
Daemon
Phantascope
After running these concepts through a series of evaluations that determined how well they fit into our design principles, we landed on the following four metaphors to develop further:
Constellations
The menu would appear as a small map of all available applications and options. This would follow the user around just above their regular field of view. A highlight would appear on the edge of the display to give peripheral awareness to the location of the menu.
Multi-Tool
The user moves the totem left and right to select a different totem/application. The user then taps the select or trigger button to select the totem. When the user selects the totem the menu will collapse and the totem “powers-up” as the app is loaded.
Phantascope
The menu and application controls appear either on totem, or on real-world objects as linear menu buttons. The totem can be attached to props such as guns or pens for enhanced input options.
Portals
Applications and content appear within bubbles of action, or Portals. These portals expand and contract to reveal applications, menus, and notifications.
The Situation
How might we make experiences more immersive with a limited FOV?
Narrow Field of View Workarounds
Screen Fade
A surface-based app only ever fills part of the room and should fit entirely into the field of view. These can be windows into other worlds either floating or bound to a single plane.
Edge Fade
Fade the edges of the screen to help blur the physical and digital to create a true mixed reality.
Simulated Peripheral Vision
For objects outside the field-of-view we have abstract virtual reflections of those objects, similar to glows on the edge of prescription lenses. Audio can also be used for indicators.
The Situation
How might we ensure experiences utilize the user’s real world most effectively?
Application Types
Full-Room
A full-room app portal fills the entire room when it is open. It can use all the planar surfaces and objects in the room. Objects within these apps should attempt to fit within the field of view.
Surface-Based
A surface-based app only ever fills part of the room and should fit entirely into the field of view. These can be windows into other worlds either floating or bound to a single plane.
Widgets
Widgets, unlike apps, can support more than one running at a time. These can be any shape and pinned to the world.
The Situation
How might we find utility and meaning in 3D internet browsing?
Browser Value Propositions
3D Object Extraction
Pull 3D objects directly out of the webpage and scale them up to lifesize within the user’s physical space.
Persistent Spaces
Deconstructed websites can be spread out within a user’s physical area and grouped by content for persistent go-to content.
Seamless Overlays
Additional visual information is layered in Z-space on top of core contrent for a richer experience.
Magic Leap’s mixed reality (MR) web browser needed to break free of the flat screen and live entirely in 3D. At the time I began, there was nothing else like this anywhere, so this was starting from scratch. As I worked through this challenge I kept asking questions:
Why would I ever browse with this instead of my phone or computer?
What does it mean to browse in 3D? What is meaningful about it? Are there reasons to do this beyond a “cool” factor?
A major limitation was the restricted field of view (FOV). Contrary to popular belief, the headset does not provide wall-to-wall coverage for total immersion. Instead it’s more like looking through a porthole. This limitation inspired ideas to pull 3D objects out of the web into the physical world. Users can set objects at any size and location to completely customize their space.
I also explored little magical pockets around a user’s space containing minimized browsing sessions. Quick and rough animation prototypes constructed in Maya.
The persistent world space presented the possibility of fully deconstructed websites and live experiences to be as spread out within the user’s space as desired. Each design needed to work within the “porthole” of the FOV.
Much of the project required visual explorations that were closely tied with UX considerations. Quick rough paintings informed various motion studies and prototypes to explore this hidden world that only comes out to play in the Magic Leap One. Deep knowledge of the UX goals stemming from months of deep UX and interaction design work guided the visuals.
And because the UI literally lives in the real world, it only made sense to explore unique material, light, and color combinations (3D models and form design by Kenneth Tay).
The Situation
How might we bring people together without external cameras and sensors?
Social (Avatar Chat) Design
Featureless but Friendly
In order to remain authentic, show only the things the system knows for sure are accurate: controller hand position and head position. Faded areas such as lower extremities and arm without a controller indicate the system does not know what the user is actually doing there.
Serve Up Emojis
Grab and throw emojis, GIFs, and other forms of expression, right from a silver platter within the space.
Electric Hi-Fives
Connect with other users across virtual space in ways that mimic physical, such as hi-fives or handshakes. Visual effects and haptic feedback let the user know they’ve “connected” in a delightful way.
Magic Leap’s Social app had lofty expectations of bringing people together in ways that were as near to physical reality as possible without external cameras and sensors. Note: this main banner is the final avatar chat marketing image - I did not create it.
The lack of external cameras and sensors created a huge challenge in terms of accurate representation of the user in 3D space. Cameras can only see the eyes in the device. They don’t see the remaining facial or body movements. The controller informs the system of the general position of the single hand, but it has no idea what the other arm or legs are doing. One of the core design principles was remaining true to the medium and not misrepresenting the user’s intent. This led to explore faceless and bodiless concepts that emphasized only what the system could detect without feeling creepy.
This is the current iteration of the avatars in the system. Note the single hand and faded lower body. Also note: this is a Magic Leap marketing image - I did not create it.
Result
These designs empowered the successful launch of the Magic Leap One.
The designs shown here were used to develop company-wide design standards to help guide various teams in simultaneous development of core apps and experiences. They also helped guide the visual design and rollout of the final product.