~ 6 Minute Read.
Today I had my final presentation of a virtual reality escape room project I did at the University of Konstanz. Together with Daniel Bogenrieder and Roxanne Low, we built this project in the Unity Engine, using a variety of technology: Leap Motion, the HTC Vive Pro, redirected walking, text to speech via IBM Watson, 3D scanning and 180° images.
The goal of our escape room was averting the user’s expectations: provoke the player to free himself of self imposed restrictions that s/he assumes are mapped from reality. That ranges from weight of objects to walking through walls and expanding humand capabilities.
I was lucky to have Daniel and Roxanne on my team since with the projects at Vhite Rabbit and my job at KNIME I was not fully committed to the project and they really did the bigger portion of work here. I will highlight some special features of the project here anyway and make sure to give credit where credit is due.
At university we have a room called the “Media Lab”. It’s a room packed full with fancy technology: full-body tracking setup, Vive, a huge multitouch screen and powerful computers.
We came up with the idea of using the Vive Pro augmented reality (passthrough mode) to be able to overlay some “perfect augmented reality” on that. It would also allow us to switch to full VR mode and use a VR headset within VR! 1
A quick test of the passthrough showed that it wasn’t very immersive, though. The resolution of the cameras does not seem to be sufficient to match the resolution of the Vive’s display.
Instead we opted for replicating this room in virtual reality, doing a full 3D scan of it and matched the tracking region to the real-world tracking area.
The setting then was that you locked yourself into this room, because you were working on an assignment for too long. Now you have 15 minutes left to hand off the assignment and need to escape. All you got is an assistant of which you don’t really know if she wants to help you or not. We called her “evil Corta[…]” (you get the idea).
I’m absolutely blown away by the opportunities given by current state of technology. With Web APIs you have a wide range of services that offer anything you can dream of.
In this case one of the first things we implemented after scanning the room was “Text to Speech” via the Watson API and their Unity SDK which provide a free contingent of characters you can send to the API (without requiring a credit card, which I don’t own).
We quickly approached this contingent so I implemented an automatic caching system that wrote out the received audio as wav files and used that whenever it was available.
Here’s a video that should give you a rough impression of how all of that sounds like:
Luckily Daniel had some experience with 3D scanning. After many attempts with different devices— Project Tango, Stereolabs’ ZED tracker, Structure Sensor and the HTC Vive Pro—we finally managed to get an acceptible scan with Project Tango.
It turns out that most of these more affordable scanning technologies work very well with objects, but have problems with scanning an entire room in one piece. Problems ranged from even getting a stable connection to software just not being able to handle the amount of data.
In the end I cleaned up the room scan in Blender of which you can see a part as a timelapse here:
This took me a lot of time and I had to cut off working on it at some point, basically resulting in a roughly remeshed room with separated objects and a texture that looked like the room survived some fire incident.
The first image from left top to bottom right is the original scan. It still has missing portions and very uneven surfaces. The second image is the cleaned up version and the third is a bunch of separated objects, partially remeshed, partially left as is.
Since I am especially proud of it, the chair deserves its own image (it alone took me 1.5 hours).
Since there is a lot to cover and my posts remain daily, I split the documentation of this project over multiple posts. Next up tomorrow is our use of redirected walking!
Written in 45 minutes, videos and images in 120 minutes edited in 10 minutes.