Mad Sand is a game taking augmented reality gaming to a new level, allowing you to shape a digital world in real time through a physical sandbox. The game is an asymmetric cooperative game, where one player (“the Sandmaker") shapes the sand landscape to help the other player (“the Penguin”) to collect sweets. Sweets can appear floating high up in the sky or deep beneath the earth, thus the Sandmaker must change the landscape so that the Penguin successfully can grab the sweets. To help the Sandmaker see what's going on an augmented reality experience is also provided through a projection of the game world onto the sand.
The goal of this project is to show the potential of how advanced interaction can combine the digital world with the real world. Interesting uses could include map making for games or sculpting of 3d digital objects, or even training simulations for archaeology.
Mad Sand is a project within the course Advanced Graphics and Interaction 2015 at at the Royal Institute of Technology (KTH), in Stockholm Sweden.
This project was sponsored by Epson.
The digital game world is created in real time using depth data from the Microsoft Kinect™, with Unity 5 as game engine. Microsoft Kinect™ sends out invisible infrared light and measures the time it takes for the light to reflect back into its camera, calculating the distance between the object and the Microsoft Kinect™.
This depth data is noisy, and must first be preprocessed before it can be applied to the in game terrain. This preprocessing proved to be a big challenge, since too much smoothing makes the game lag, while too little while make the terrain constantly flicker. With preprocessing complete, the depth data is converted to a depth map that is then used by the in game terrain.
Finally the augmented reality experience is provided by a projection of the current in game world onto the sand, using an Epson projector. A challenge with this projection setup is that it is very sensitive to light pollution, needing both protection from other light sources and a well contrasted image to project. Further aligning the projected image with the actual sand landscape is hard, especially if you want real time performance.
Character animation, 3d modeling, C#, Microsoft Kinect™, real time rendering, projection on non-flat surfaces, Third person camera, Unity 5, User testing, Git.
Multiplayer: allow for more than one penguin at the time, perhaps even in a competitive game mode?
Dual projection: project the game world onto the sand through multiple projectors, thereby avoiding shadows from e.g. hands, and allow for a more precise projection of objects.
Who are we, the mystery people behind the sandbox? Continue reading and you might find out.