ETH is dancing with Google Tango

A smartphone that can navigate you through a shopping centre and straight to the cereal aisle. A device that shows us how a sofa we have not yet purchased will actually look in our living room. Technology that can lead the blind much in the way that guide dogs do today. All of this could soon be a reality. A project group at internet giant Google is working with hardware manufacturers, public research institutions and others to develop this type of mobile device along with the necessary software. ETH professors Marc Pollefeys and Roland Siegwart and their staff are playing a major role in this initiative, codenamed Project Tango.

The key feature of the new device is its ability to capture the three-dimensional environment in detail and in real time. To do this, it not only has a camera, accelerometer and angular rate sensor like any other smartphone, but is also equipped with a second built-in camera and an infrared light source. The latter projects a pattern invisible to the human eye on to the surroundings that is then captured by one of the two cameras. Using this information and the data from the acceleration and angular rate sensor, the device can then calculate detailed spatial information.

Google has already produced prototypes that scientists from ETH Zurich are using for their research and to improve their algorithms. Google has announced that the first device should come on to the market next year; reportedly, it will be a tablet running the Android operating system.

Virtual and real world combined

“The possibilities of this device and the 3D data it delivers are limitless,” says Roland Siegwart, Professor of Autonomous Systems and Vice President of Research and Corporate Relations at ETH Zurich. “An incredible number of apps could be developed based on this technology. It remains to be seen what these will be.” The device may also open up entirely new possibilities for applications such as computer games. For example, the virtual content of games could be combined on the screen with the real surroundings captured by the camera. Gamers could use their mobile phones to navigate their way through virtually enhanced real environments, with virtual characters and objects moving with them on the screen. This combination of virtual and real worlds might also be useful for interior design visualisations.

Users can also use the technology to create their own 3D maps or download maps created by others and then use them on their devices. The device compares the surroundings with stored map data in order to determine the location, enabling GPS-like navigation even indoors. In contrast, GPS does not work indoors because the necessary satellite data cannot be received.

Advantages over GPS

Simon Lynen, a PhD student in Siegwart’s group, is involved in developing the map function for the Tango device. Using a prototype device mounted on a bicycle helmet, he has mapped the streets of Zurich with this function, showing how the technology can also be used to navigate outdoor routes over many kilometres. “Our technology offers a clear advantage over current GPS technology outdoors too. GPS only provides the location, whereas our technology detects the viewing direction of the cameras,” says Lynen. By way of example, this enables information to be displayed depending on the viewing direction. In addition, the new technology is expected to 10 times more effective than GPS at determining location, at least within the limited scope of existing maps.

The ability of the device to determine the location in just a fraction of a second after starting the device (or whenever necessary) via comparison with the stored map is also based on ETH expertise. PhD students at ETH are also currently working on a solution that allows the device to capture 3D data from the surroundings when it cannot process the projected infrared pattern – for example, due to intense sunlight. “To do this, the device uses a sequence of ‘normal’ camera images to capture the depth data much in the way we do by moving our heads,” explains Marc Pollefeys, a professor in the Department of Computer Science.
Error correction

ETH Zurich researchers are also interested in how to correct slight location errors that may occur due to measurement inaccuracies. These accumulate when a user travels for extended periods through corridors or on roads, and can result in a difference between the calculated location and the actual location of the mobile device. ETH researchers are now working to correct these errors by making use of the image data stored on the mobile device: if the user returns to a previously visited location, the device will detect this and reset any possible deviation to zero.

Pollefeys’s and Siegwart’s groups have already acquired considerable expertise through collecting geometric spatial data from visual information in the course of projects such as sFly  for autonomous navigation of flying robots and V-Charge  for driverless car parking. Pollefeys and his group have also developed an app that transforms a smartphone into a 3D scanner . “Our proven expertise in this field is probably why Google approached us to work with them on Project Tango,” says Siegwart. Pollefeys adds: “The basis of all these projects are related. Thus, our staff can contribute the knowledge acquired from past projects to Project Tango.”