ARchitect: Building Interactive Virtual Experiences from Physical Affordances by Bringing Human-in-the-Loop
The Hong Kong University of Science and Technology
Automatic generation of Virtual Reality (VR) worlds which adapt to physical environments have been proposed to enable safe walking in VR. However, such techniques mainly focus on the avoidance of physical objects as obstacles and overlook their interaction affordances as passive haptics. Current VR experiences involving interaction with physical objects in surroundings still require verbal instruction from an assisting partner. We present ARchitect, a proof-of-concept prototype that allows flexible customization of a VR experience with human-in-the-loop. ARchitect brings in an assistant to map physical objects to virtual proxies of matching affordances using Augmented Reality (AR). In a within-subjects study (9 user pairs) comparing ARchitect to a baseline condition, assistants and players experienced decreased workload and players showed increased VR presence and trust in the assistant. Finally, we defined design guidelines of ARchitect for future designers and implemented three demonstrative experiences.
~6 minute read
Modern commercial Virtual Reality (VR) systems (e.g., Oculus Rift and HTV Vive) commit to achieving full immersion in the virtual world yet come with the cost of losing spatial awareness of physical surroundings. Since virtual and physical realities intrinsically overlap in space, movement and actions of users in the physical environment directly affect their experiences in the virtual world. For example, bumping into physical obstacles may be dangerous and contacting objects in the physical world that are not rendered in VR may lead to breaks in presence from confusion. Hence, users are often restricted to a small confined area as to not encounter with physical objects in surroundings.
However, prior work has shown that real walking in VR strengthens virtual presence. In addition, prior research has also found that receiving haptic feedback from touching physical objects registered with virtual proxies increases presence in the virtual world. In our work, we aim to utilize the merits of physical walking by segmenting a walkable area and the blending physical objects as passive haptics to build an immersive interactive virtual experience.
We present ARchitect, a proof-of-concept prototype which facilitates the flexible construction of an interactive virtual experience that allows both real-walking and physical interactions. In our work, we exploit the asymmetric dynamic between two users: one as the "Assistant" in AR and the other as the "Player" in immersive VR.
We designed ARchitect with three central design objectives:
- Build a system which allows flexible customization of an interactive virtual experience that reflects the physical environment in both geometry and interaction affordances.
- Ensure that the process of building the virtual experience is straightforward and requires minimal effort through several automated functionalities to aid the Assistant.
- Promote easy adoption of our system for the public by deploying it on a mobile platform. Hence, users are alleviated from the need of having a dedicated tethered VR system.
According to our design objectives, the following is our system implementation.
As the Assistant walks through the environment, the mobile device scans the scene and coarse geometries such as horizontal planes are highlighted to aid the Assistant. The Assistant, who sees the physical world, then superimposes physical objects with virtual proxies in consideration of affordances and theme consistency to design a virtual world. To aid the Assistant, our system also recommends virtual proxies based on interaction affordances of in-situ objects.
After the Assistant finishes mapping the virtual world, the mobile device is switched to VR mode, encased in a Google Cardboard, and given to the Player. The Player may then experience the virtual world, built by the Assistant using the physical world as a template, with physical interactions and real walking.
We conducted a within-subjects user study with 18 participants (6 female, 12 male) aged from 20 to 30 (mean=22.39, SD=2.79) to evaluate participants' experience of presence (SUS questionnaire), trust (Dependency component of Trust in Close Relationships questionnaire), and workload (NASA TLX questionnaire) between ARchitect and a baseline. More specifically, we measured the presence, trust, and workload of the Player and the workload of the Assistant after experiencing each system. In the baseline condition, the Player was required to complete the same tasks without interactable virtual proxies embedded using ARchitect. Instead, the Assistant guided the Player through vocal instructions to interact with the physical environment.
User study results were analyzed through a one-way repeated-measures ANOVA. Generally, participants reported a significantly higher sense of virtual presence, a significantly higher sense of trust, and a significantly decreased workload when using ARchitect compared with the baseline.
We further implemented three example experiences to demonstrate ARchitect’s ability of supporting the Assistant from three directions: (1) repurpose design, (2) level design, and (3) personalization through modular design.
LavaEscape (Repurpose Design)
In LavaEscape, we explore the benefit of ARchitect for in situ repurposing of everyday objects into game elements, without the need for specialized props to fit a storyline. The experience also examines the Assistant’s ability to select appropriate physical objects for mapping (e.g., stable objects to be stood on) as well as the ability to accurately fine-tune the size, orientation, and scale of virtual proxies for a reality-based virtual environment with strict mapping requirements.
Maze (Level Design)
In Maze, we explore the use of ARchitect for level design while reusing the physical environment. We explore the usage of fake barriers and obstacles to manipulate the concept of space in the virtual world under an asymmetric setting. Hence, with ARchitect, the virtual world may be designed to have greater variety and be far more complex. An interesting area for further exploration may be its usage for designing redirected walking experiences where Players explore a virtual world that is deceivingly larger than the physical configuration offered in reality.
Sandbox (Personalization through Modular Design)
In Sandbox, we illustrate the ability to create personalized experiences by exploring the modularity of ARchitect. Sandbox is a testbed for Assistants to establish their own game theme and configure their own set of virtual proxies for different components of the virtual world. In Sandbox, we also explore the possibility of incorporating a physical controller into the virtual experience, perhaps even extending to haptic actuators to blend active haptics.
This work is partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China under Grant No. T44-707/16-N.