VR Experience Design Document

Here is some documentation for my recent work at Snap involving the World Mesh, Custom Locations, and the Custom Location Creator Lens:

9/20/23: live stream I hosted on World Mesh 2.0 release
9/13/23: blog post on tech releases I worked on for World Mesh and Custom Locations
7/18/23: live stream I cohosted on City Scale Location AR technology in Lens Studio
6/13/23: live stream I hosted on Custom Location AR technology in Lens Studio
1/24/23: live stream I hosted on best practices for location AR in Lens Studio
Video walkthrough of the Creator Lens is available on this documentation page
UX research case study on Creator Lens

Title: C-Spresso

Platform: Untethered VR Oculus Quest

Genre: Educational Adventure

Target: Middle to highschool students

The world of C-Spresso is one made up of 1’s and 0’s, candy, tea cups, coffee mugs, and space travel. It resides right in the center of the Milkyway, where inhabitants from all over the galaxy come together to share a pot of tea and enjoy sugary treats. It is also a vessel that is controlled by computer science and where you may learn binary math and coding. In this immersive experience, you find yourself inside a spaceship that has been damaged during an intergalactic space storm. Here, you’ll meet the ship’s assistant, Eggbert (but you may call him Eggy), a little egg timer who grants you the quest to help repair the C-Spresso so that it can return home to the center of the Milky Way, in Dessert Star.

Executive Summary

C-Spresso was designed to encourage learning and exploration of Computer Science principles. This is a learning application as well as a gamified experience where the user learns how to solve different math equations while playing and moving around. With the Oculus Quest, users with multiple learning styles can benefit from the 360, interactive environment. Spatial and kinesthetic learning styles are central to the lessons as the user has free roaming abilities in VR and is constantly pulling levers, grabbing objects, and shuffling objects around. The experience is designed to present what is often perceived as CS in a playful, appealing light where the user can more easily make a connection to the material, rather than passively listening to a lesson or reading a text book. C-Spresso is meant to be an inclusive, accessible learning tool so that females, underrepresented minorities, and even those with attention issues can have another option, another platform to learn CS and mathematics. Choices regarding the story, aesthetic, and interactions were made in regards to creating an experience that appeals to a wide demographic. It is made to offer a chance to many who may not otherwise learn CS and mathematics in a traditional classroom setting. We believe this is the case with C-Spresso because the core mechanics and gameplay utilize spatial learning and physical movement and engages the user in the lesson through storytelling and characterization. In this experience, users learn these complex topics while making candy, playing with sugar goblins, and repairing a candy spacecraft.

Design Videos:

Lecture videos from recent Educational Software Design class that covers design of binary counting and bubble sort level of CSpresso:

Unique Selling Points

VR experience design for Oculus Quest
Spatial learning experience in untethered VR

Design Objectives

Increasing engagement factors by appeal to both halves of the brain (logical and creative halves)
Free roaming abilities- serves the purpose of getting kids moving as opposed to sitting for hours, listening to lectures
Include multiple learning styles: kinesthetic, spatial, visual
To build a sense of magic, fun, and exploration through narrative, characters, and interactions
Create enthusiasm for learning CS and math principles

Learning Objectives

This experience must be accessible, engaging, and have an adjustable pace to suit each user.
Concepts must be taught through play and fun
Experience should not appeal only to those who have an understanding of CS
Environment and aesthetic must provide a safe space for users to make mistakes, explore, and learn about STEM

Objective is to expose students to STEM and computer science in an engaging and immersive environment. Due to the Quest’s untethered and free range of movement feature, educators and researchers believe this CS application could benefit students who are visual and spatial learners. Spatial Learning refers to an individual’s ability to perceive, analyze, and understand visual information in the world around them. Most US schools use a teaching method called Linear-Spatial Thinking, which does not benefit students with hands on or visual learning styles. To ensure the efficacy of this CS experience, we have researched other methods of teaching CS concepts.

CS Unplugged is an organization that provides educators with free, open source, material for CS education. The learning activities are constructed so that they do not use a computer, but instead require physically and visually engaging group activities. We have chosen three activities from CS unplugged to use as models for our VR application:

Binary Cards
Bubble Sorting
If Statements, Loops

The player must go through the experience in the following order: be introduced (to the concept)>practice>learn>demonstrate (a learned understanding of the concept)>apply (the learned skills in next level)

Game Objectives

The objective of the experience is to restore the C-Spresso back to repair so that can go back to its home in the city of Dessert Star.

In the first level, the objective is for the user to gradually complete each binary math exercise as Eggy brings her/him through the enclosed dome space. Eggy grants the player with the quest to help him repair the C-Spresso by getting all 5 stations running again. The player does this by going through an assembly line flow where he/she must go to each of the 5 stations in a clockwise direction. Each station is one element of a manufacturing stream of parts needed to repair the ship.

Gamifying the Binary System

The binary system contains numbers expressed in base-2 numeral system which uses only two symbols--0 and 1. This is shown in the game as levers which the user pulls down or pushes up in order to change the number from 1 to a 0 or vice versa.

In CS Unplugged binary math is represented with cards, each with two sides; one side with a number dots and the other side with the symbol representation of that number of dots. The user flips a sequence of cards until they add up to a desired amount. We created our lever system based off of this mechanic, where the binary card takes the form of a 3D lever which has a corresponding dial with a 0 or 1 on it. The user does the same action of “flipping” the value from 0 to 1 (or vice versa), by moving the lever into one of it’s 2 positions--up or down.

Core Mechanics of the Stations

The design of the stations (both functionality and aesthetic), is inspired by kiosks and consoles. The reason for this is because while designing this experience, we realized that the user would be manipulating objects to get a certain outcome, much like interactive objects in the real world that we regularly use: ATM’s, parking meters, self checkouts and arcade machines.

Number Station

The Number Station is designed to feel similar to an arcade kiosk, with levers and a display screen. There are currently 3 levers, each representing a number of a binary math equation. The equation can be viewed on the display screen, where the user can view their actions when pulling each lever. For example, when the user is prompted to make 4 red circles, he/she is directed to the number station where he/she will pull each of the three levers to get the total of 4. The levers would have two positions so as to not distract the user from the 0 or 1 action. There would also need to be a display at this station so that the user can see what he/she is doing. There are 3 levers, which are represented below:

UX/Interaction Design

Each interaction in the experience had to meet the following usability goals:

Learnability- The system requires the user to repeat similar actions of reaching>grabbing>pulling (or pushing up), throughout the experience. Each station works the same way, in how they are activated by pulling levers. Any other ways of activating the stations would require the user to learn more actions and increase cognitive load. This must be reduced as the user is already trying to retain CS knowledge throughout the experience.

Memorability- The system must be designed so that the user does not have to be overwhelmed with re-learning how to bring about a desired outcome or stress over trying to remember how the system works, each time he/she approaches the next station. This experience has to remain a relaxing, yet engaging one to allow the user to feel focused and comfortable while learning binary math.

Accessibility- The experience must be designed for both seated and free roaming play. Users who have the ability to move around and have a 20 x 20 ft space can choose to play the free roaming version. Users who have limited mobility will not be excluded from the experience and can select the seated play version, which can be played while sitting.

To stay true to these requirements, the stations must be designed with similar interfaces, actions, and the information on those interfaces must be in the same areas.

One example of how to do this is by placing the display or input screens on each station above the lever dials, at eye level. The lever systems all have dials with the corresponding number located right above each lever. Based on these design choices, the user should be able to remember how the system works with each station. Each lever connects to a dial that displays their output and can be changed by simply pulling down or pushing upward.

There will also be an NPC guide to reinforce interactions learned at each station, to help solidify this interaction knowledge.

Types Of Interactions

Macro Interaction Designs- From the perspective of the system

Processing user input, guiding/prompting user through navigation and providing help

Middle Interaction Designs- From the perspective of the stations

Processing user input, provide lighting, sound, and visual cues to communicate to user during interaction

Micro Interaction Designs- From the perspective of the individual elements (levers)

Must function with minimal effort from the user, must indicate active and inactive states ( lever handle turns gray when inactive and glows green when active)

Micro Interactions-Categorized by these properties:

Triggers
Cues
Behaviors
Actions
Modes
Mode selectors
Feedback
System

Once these micro interactions were sorted out, the designers decided to make the stations “single mode.” To keep these processes and design within reasonable scope, the elements in the experience mono mode

Troubleshooting/Risk Assessment

Designing an experience that remains true to the usability goals of memorability and learnability, requires a thorough understanding of what possible difficulties the user may encounter. Identifying these potential problem areas provided a framework for an interaction design with the best possible outcomes:

User Scenario: User Gets Lost

We thought of what might happen if the user did not understand the flow of action. The resolution would be a timing function (estimated 30-60 seconds) where the user would be prompted if he/she did not get to the next station within a designated time frame.

User must know if going the wrong way

SOLUTION: The resolution would be a timing function (estimated 30-60 seconds) where the user would be prompted if he/she did not get to the next station within a designated time frame.

OBSTACLE: When the user arrives at the #S. The user needs to somehow see that the #Station action is directly showing up on the OPS screen.

SOLUTION: Cue user to look over at OPS? Have some sort of guide that tells them about the connection between the #S and OPS screen?

Prototyping-Outcomes and Changes

C-Spresso has gone through low and high fidelity prototypes for us to gain an accurate perspective of the gameplay in physical space. The size of the play area was set at 20 x 20 ft so that the user can take advantage of the free roaming abilities and physically walk in the loop structure of the first level. We set up a 20 x 20 ft space where all 5 stations were placed in the clockwise, concentric circle position and tested out the interaction flow. Doing this gave valuable insight into the feel of the experience, even in this low fidelity state. Each play testing of the prototype revealed new information on what was missing or could be improved:

Paper Prototype

Problem: Need directional cues

Solution: NPC guide, floor patterns in concentric circle layout

In the first prototype, the stations were built like small kiosks and they were placed very close together. After going through a rough floor plan of the experience, the layout of the kiosks was modified. The space seemed really large, so it was concluded that this would change as a result of adding more assets which would populate the scene, such as the dome enclosures that would be around each kiosk. A cylindrical wall was added around the entire floor space and this drastically made the space feel less spread out and empty. The play test of the digital prototype revealed new improvements needed:

Digital Prototype

Problem: Space is too large, empty

Solution: Enclose inside a dome structure with opaque wall textures, move stations closer together

Throughout all stages of prototyping and playtesting, it was clear that there must be something in charge of prompting and guiding users throughout the experience as well as communicating to the user if making a mistake. A few previous ideas included a character/host, or voice overs coming from each station. After receiving user feedback, we decided on the NPC character guide, Eggbert.

Space & Ergonomics

Considering the ergonomics of these kiosks was another challenge because this experience is designed for both middle schoolers and adults, so the height to build these kiosks is based on that of most other consoles such as ATM’s and arcade kiosks.

The kiosks will be designed to be ergonomically sound. This means that the dashboard with the controls need to be placed at a level where the user does not need to bend down or tilt their head up. The width of the dashboard, UI, and display also must be designed to be within scope of the user’s vision. If the kiosk is too small or too large, it will not be comfortable for the user to interact with.

Binary Math Level 1- Runthrough :

In the first level, the player engages in a binary math lesson. The NPC guide, Eggy, shows the player around the dome space, lined with 5 stations. Each station is a new binary math concept and part of a binary math equation.

The loop begins with the Task Station, where the user is given an exact description of what is needed. The player must go through each station, following the clockwise direction in which they are placed. Each task attaches to the next task, so that the player is creating an object with specific requirements such as number, color, and shape of objects requested by the Task Station. The player must complete a task (a binary math equation), at each station so that by the time he/she reaches the Output Station, he/she can submit their result.

Story Overview

All sweetness in the galaxy comes from one secret location, smack dab in the center of our Milky Way. If you are ever lucky enough to make it there, you will realize that the very air you breathe tastes like the inside of a bakery or maybe a candy store. At that point in your journey, you will know that you are approaching the hidden city of Dessert Star . If you’ve ever wondered where the very concept of sweets and treats originated, it is here. The atmosphere of Dessert Star is completely made up of a chemical compound called ethyl formate, which gives earth the scents and tastes of everything sweet, from fruits to candy, and even flowers.

The C-Spresso is Dessert Star’s most iconic and significant space station. Here, all of the galaxy’s sweet treats and confectionary delights are manufactured and distributed to all inhabitants within the Milky Way. This helps keep the peace within the galaxy, keeping everyone happy and peaceful, with sugary treats, nice hot coffees, and even tea time...especially tea time. All within the galaxy love coming to the city of Dessert Star, to unwind from a hectic work day, or to share a pot of tea with other nearby, intergalactic friends. While Earth is within the Milky Way, it is just too far away to visit in time for brunch or tea time, so the C-Spresso makes its rounds along the Milky Way, doing air drops of coffee beans, sugar cane seeds, and tea leaf plants for the little Earthlings to harvest and distribute among themselves. The next time you find yourself enjoying a candy cane, liquorice, or hot cocoa, you could appreciate this magical place far away in the center of the Milky Way, that is rumored to taste like raspberries...and you will know why!

It has been this way for centuries, and has kept all within the galaxy happy and full of cheery energy. Unfortunately for them, an asteroid has just torn right through the center of the Milky Way, splitting the docking station where the C-Spresso recharges between rounds and hurling it far off into space. The inhabitants of the galaxy have since been in a frenzy, trying to locate the ship and bring it back to Dessert Star.

Someone brave will have to go out into the inky blackness outside the Milky Way, find the C-Spresso, and bring her back to Dessert Star, so that everyone’s lives can go back to normal. Until then, the inhabitants of the Milky Way will be dreaming about when their ship will be back so they can warm up, after a much too long period without delightful hot beverages or sweets!

Demographic Research

In preparation for developing this experience, we did research into the demographic we were designing for; grades 5-12, male and females. We researched highest grossing films, games, and series within children’s entertainment. After spending time gathering notes on what titles drew the most interest from all gender groups, we began to find a thread of commonality to start following. Game titles such as Kirby, Mario Odyssey, Tynker games, and Zelda shared similar color palettes of bright greens, golds, and oranges. Shapes were large, simple, and rounded in form (with the acception of Zelda, which was more sophisticated in visuals with more realistic forms), and the overall atmosphere within each world was cheerful, with light-hearted music. After researching film and series titles that appealed to many different demographics, we first, were interested in titles that had a timeless quality--a story that did not fade with time but still draws the attention of new generations. Some of these titles were Willy Wonka’s Chocolate Factory, Alice in Wonderland,The Wizard of Oz, and Dr. Suess stories--all share this whimsical silliness that seems to leave a lasting impression and fondness with their audience. As artists, designers, and storytellers, we wanted to create an experience that was designed with these influential titles and stories as core inspiration.

The first goal of designing this application was to take full advantage of this new untethered feature, which would offer users another level of immersion as they physically move through virtual space. The virtual space is organized into a concentric circle, lined by consoles and kiosks, all chronologically placed into a clockwise direction. This design follows the order of interaction from start to finish, in one cohesive loop. This ability of the Quest allowed the team to incorporate spatial learning methods into the VR application, as the user walks from station to station, retrieving, placing, and reaching for objects related to the lesson. Adding the element of spatial learning into the experience opened many exciting opportunities for the designers to explore while creating the environment, narrative and game mechanics.

Art bible

The “style guide” portion of the writing. Guidelines for all other artists and designers that set the look and feel of the experience or game.

Concept Art Overview

C-Spresso is a name referencing the design of espresso and barista machines, which the quirky designs of the kiosks and consoles resembled. Once this connection was made, the visual design began to take form, as designers researched various barista and confectionery machines such as gumball, cotton candy, and espresso machines. The creative direction for the experience is meant to maintain a light-hearted and fun mood, while users are learning complex and often intimidating math skills. The team hopes to immerse the users in a safe, engaging environment where he or she may explore, learn, and make mistakes, while having fun and playing.

Character Line-Up

Player Characters/ Avatar

The player takes the role as a wanderer who has stumbled upon a strange spacecraft which they learn is the C-Spresso. They learn this from the NPC and guide, Eggy, who is a resident on the spacecraft and grants the player with a quest to help him fix the ship so they may return home to the center of the Milky Way in Dessert Star.

The player will have the ability to physically walk around the space as he/she is interacting with the 5 stations within the level. Currently, the user has the ability to grab, hold, and place objects, all by squeezing the triggers on the hand controllers.

Eggbert aka “Eggy”

We designed the experience so that the user would interact with as few 2D interfaces as possible. Eggy would be serving as not only a guide, but as a “start” button for the beginning of the experience when the user would be prompted to give him a high-five, when ready to begin. Eggy is also the guide for the player throughout the experience. Eggy serves many different roles in the first level of the experience. He is not only a friendly NPC character, but also part of the system mechanics itself.

Roles:

Navigation
Prompting
Help
Guidance
Interaction

Animations:

Bottom half spins so he can hover and fly around the station
Simple 2D facial animations
Oven mitt hands move around close proximity to his body and can bend, fold, and grab things

Sounds:

Child voice
Beeping
Spinning
Humming

Tea Cup Gremlins

Tea Cup Goblins spawn when the user stacks found tea cups onto each other in a certain order or position. These creatures serve the purpose of encouraging the user to explore the environment and be creative with the objects around the station.

The Tea Cup Goblins have a stronger purpose when the user starts learning about exponential numbers. Here, the user would learn about exponentials by placing a certain number of objects into a container and stacking these containers. The user will create different Tea Cup Goblins depending on the combination of tea cups that they stack onto each other.

*refer to exponential chapter in math book

Sugar Goblins

Sugar Goblins are a pesky species who live in the Milky Way and thrive on sugary treats made from the C-Spresso. They are high energy creatures and only their diets consist of sugar and caffeine. Sugar Goblins need sugar and caffeine throughout the day otherwise they start to crash, which could be dangerous to the player.

Crash Phases:

Phase 1- Moody, irritable, unfocused

Phase 2- Aggressive, combative, hoarding, territorial

Phase 3- Slow down, sleepy, cold

Phase 4- Lose color, turn to gray, immobilized

It is crucial to keep the Sugar Goblins well fed, while also making sure that they don’t eat all of the treats on the C-Spresso. If the player comes in contact with a Sugar Goblin who is in any of the crashing phases, he/she must intervene before it reaches phase 2, otherwise, the player will be in danger of

The Sugar Goblins are a necessary evil because they help to spread the ethyl formate around the galaxy, much like hummingbirds and bees help to pollinate flowers. They also are essential and keeping the level of ethyl formate even.

Future Plans:

Avatar: we would like the player to be able to choose his/her own avatar from a selection of different space suits or maybe just some kind of space helmet since the air there is ethyl formate and not oxygen. This would give the player a better connection into the actual storyline of the experience.

Eric Nersesian

eric.nersesian@njit.edu