ARchitect: Building Interactive Virtual Experiences from Physical Affordances by Bringing Human-in-the-Loop
The Hong Kong University of Science and Technology
(a) ARchitect allows physical objects to be mapped to virtual proxies offering matching affordances (e.g., both "chair" and "tree stump" afford a "sitting" interaction). A physical scene (b) may be translated using ARchitect’s user interface (c) to an interactive virtual experience (d).
Abstract
Automatic generation of Virtual Reality (VR) worlds which adapt to physical environments have been proposed to enable safe walking in VR. However, such techniques mainly focus on the avoidance of physical objects as obstacles and overlook their interaction affordances as passive haptics. Current VR experiences involving interaction with physical objects in surroundings still require verbal instruction from an assisting partner. We present ARchitect, a proof-of-concept prototype that allows flexible customization of a VR experience with human-in-the-loop. ARchitect brings in an assistant to map physical objects to virtual proxies of matching affordances using Augmented Reality (AR). In a within-subjects study (9 user pairs) comparing ARchitect to a baseline condition, assistants and players experienced decreased workload and players showed increased VR presence and trust in the assistant. Finally, we defined design guidelines of ARchitect for future designers and implemented three demonstrative experiences.
Video Summary
~6 minute read
Background
Modern commercial Virtual Reality (VR) systems (e.g., Oculus Rift and HTV Vive) commit to achieving full immersion in the virtual world yet come with the cost of losing spatial awareness of physical surroundings. Since virtual and physical realities intrinsically overlap in space, movement and actions of users in the physical environment directly affect their experiences in the virtual world. For example, bumping into physical obstacles may be dangerous and contacting objects in the physical world that are not rendered in VR may lead to breaks in presence from confusion. Hence, users are often restricted to a small confined area as to not encounter with physical objects in surroundings.
Current VR systems restrict the user to a fixed confined area.
However, prior work has shown that real walking in VR strengthens virtual presence. In addition, prior research has also found that receiving haptic feedback from touching physical objects registered with virtual proxies increases presence in the virtual world. In our work, we aim to utilize the merits of physical walking by segmenting a walkable area and the blending physical objects as passive haptics to build an immersive interactive virtual experience.
Method
We present ARchitect, a proof-of-concept prototype which facilitates the flexible construction of an interactive virtual experience that allows both real-walking and physical interactions. In our work, we exploit the asymmetric dynamic between two users: one as the "Assistant" in AR and the other as the "Player" in immersive VR.
Design
We designed ARchitect with three central design objectives:
- Build a system which allows flexible customization of an interactive virtual experience that reflects the physical environment in both geometry and interaction affordances.
- Ensure that the process of building the virtual experience is straightforward and requires minimal effort through several automated functionalities to aid the Assistant.
- Promote easy adoption of our system for the public by deploying it on a mobile platform. Hence, users are alleviated from the need of having a dedicated tethered VR system.
Implementation
According to our design objectives, the following is our system implementation.
We have defined the major components of ARchitect as the following: Scene Understanding, the Affordance Recommender, the Placement Algorithm, Tracking, and the Game Engine.
As the Assistant walks through the environment, the mobile device scans the scene and coarse geometries such as horizontal planes are highlighted to aid the Assistant. The Assistant, who sees the physical world, then superimposes physical objects with virtual proxies in consideration of affordances and theme consistency to design a virtual world. To aid the Assistant, our system also recommends virtual proxies based on interaction affordances of in-situ objects.
The Affordance Recommender detects predefined interactable classes (e.g., "chair" class corresponds to "sitting on" interactable).
After the Assistant finishes mapping the virtual world, the mobile device is switched to VR mode, encased in a Google Cardboard, and given to the Player. The Player may then experience the virtual world, built by the Assistant using the physical world as a template, with physical interactions and real walking.
The virtual view of the Player (a) while bending down to collect virtual mushrooms (b).
The virtual view of the Player (a) while sitting down to take a rest (b).
User Study
We conducted a within-subjects user study with 18 participants (6 female, 12 male) aged from 20 to 30 (mean=22.39, SD=2.79) to evaluate participants' experience of presence (SUS questionnaire), trust (Dependency component of Trust in Close Relationships questionnaire), and workload (NASA TLX questionnaire) between ARchitect and a baseline. More specifically, we measured the presence, trust, and workload of the Player and the workload of the Assistant after experiencing each system. In the baseline condition, the Player was required to complete the same tasks without interactable virtual proxies embedded using ARchitect. Instead, the Assistant guided the Player through vocal instructions to interact with the physical environment.
(a) is the user study setup and (b) - (g) portray participants performing several different tasks using ARchitect.
User study results were analyzed through a one-way repeated-measures ANOVA. Generally, participants reported a significantly higher sense of virtual presence, a significantly higher sense of trust, and a significantly decreased workload when using ARchitect compared with the baseline.
Means and standard deviations of presence, trust, workload (Assistant), and workload (Player) scores comparing ARchitect and the baseline. Error bars are standard errors.
Applications
We further implemented three example experiences to demonstrate ARchitect’s ability of supporting the Assistant from three directions: (1) repurpose design, (2) level design, and (3) personalization through modular design.
LavaEscape (Repurpose Design)
In LavaEscape, we explore the benefit of ARchitect for in situ repurposing of everyday objects into game elements, without the need for specialized props to fit a storyline. The experience also examines the Assistant’s ability to select appropriate physical objects for mapping (e.g., stable objects to be stood on) as well as the ability to accurately fine-tune the size, orientation, and scale of virtual proxies for a reality-based virtual environment with strict mapping requirements.
(a) In LavaEscape, the Assistant first maps physical furniture to rocks of various shapes and sizes (and heights). The Player steps onto the lowest piece of furniture (b) and continues moving onto higher furniture as the lava plane rises (c) until reaching the higher ground (d).
Maze (Level Design)
In Maze, we explore the use of ARchitect for level design while reusing the physical environment. We explore the usage of fake barriers and obstacles to manipulate the concept of space in the virtual world under an asymmetric setting. Hence, with ARchitect, the virtual world may be designed to have greater variety and be far more complex. An interesting area for further exploration may be its usage for designing redirected walking experiences where Players explore a virtual world that is deceivingly larger than the physical configuration offered in reality.
A Player attempting to locate the treasure chest (3D audio source) (a) while playing Maze. The virtual maze corresponds to the physical room configuration (walls and table) with the addition of a fake virtual wall (b). (c) shows an example completion path that may be taken by the Player.
Sandbox (Personalization through Modular Design)
In Sandbox, we illustrate the ability to create personalized experiences by exploring the modularity of ARchitect. Sandbox is a testbed for Assistants to establish their own game theme and configure their own set of virtual proxies for different components of the virtual world. In Sandbox, we also explore the possibility of incorporating a physical controller into the virtual experience, perhaps even extending to haptic actuators to blend active haptics.
A Player playing with a skateboard simulator configured with Sandbox (a) using a physical skateboard controller (b). The experience may be reconfigured to a different theme (e.g., aquatic (c) or space (d)).
Acknowledgments
This work is partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China under Grant No. T44-707/16-N.