LiDAR and virtual reality–it’s a marriage made in Hollywood.
A 1.499Mb PDF of this article as it appeared in the magazine complete with images is available by clicking HERE
Innovative Southern California startup HypeVR, a live action virtual reality (VR) technology provider, and Silicon Valley’s Velodyne LiDAR, have teamed up to inject a bit more "you are there" into VR. Soon enough, the fruits of that relationship will be coming to a multiplex, videogame or headset near you. If you’ve ever wanted to lean into a scene in full 3D, the wait is almost over.
"I first was able to experience live action 360-degree video in VR about a year and a half ago," recalls Tonaci Tran, HypeVR co-founder. "The thing I realized fairly quickly was that stitching multiple camera angles together to create 360 degree panoramic video was incredibly challenging.
"When you’re doing that stitching, you’re trying to connect different pieces of the environment," Tran says. "And when playing back video, you get all these errors where the pieces overlap. Somebody will disappear or a face will get cut off. I thought there must be a better way to do this."
From that epiphany, HypeVR was born. "I wanted to create a solution to produce seamless results when creating 360-degree VR," Tran recalls. "It was clear that we had to develop in a 3D space. And that’s where LiDAR comes in. I found Velodyne, which has a product that is portable enough, robust enough and delivers on what they said it could do. Velodyne’s multichannel HDL-32E 3D LiDAR sensor enables us to get very accurate depth data at relatively long distances and helps us to stitch images together seamlessly. It was the missing link. To make data useful and work with our method is a challenge, but making it work and integrate data points into our seamless stitching method is an even bigger challenge."
Using a proprietary, patent-pending 3D method–which fuses HypeVR’s "Hyper Stitch" algorithm with 3D mapping data from Velodyne’s 3D LiDAR sensor–viewers are able to "lean into a scene" with full six degrees of freedom within a 360-degree live action stereoscopic VR environment. HypeVR’s flagship virtual reality rig consists of 14 Red Dragon cameras and Velodyne’s spinning LiDAR sensor, which provides the ability to simultaneously capture a motion image of 84K at up to 90fps and a 360-degree point cloud at 700,000 points per second.
"The data points we rely on are dense from a point cloud perspective," Tran says. "Shooting camera images are photographic and extremely dense. The two are not same. Trying to use both and combine them, as we are, is a formidable task.
"Movies are traditionally shot in one direction, 180 degrees in front of you," he notes. "The initial rig gives us a blueprint, a foundation for capturing the highest resolution image that we possibly can for complete 360-degree coverage. That has allowed us to combine footage with LiDAR data and apply it to our patent-pending algorithm. That is what enables us to output a very `photographic’ motion image.
"Just imagine shooting on a standard depth camera," he says. "The footage simply wouldn’t work for 360 degree 3D. Everyone has an HD monitor, and 4K monitors are now affordable–they’re even at Costco. For 360 degree immersion, you need higher image quality. And for that immersion, you cannot have visual errors." Feature film work involves budgets substantial enough to ensure that kind of quality, Tran suggests.
"I couldn’t believe my eyes when we got six-degree VR working for the first time," he recounts. "The extra dimension of being able to lean into a live action scene really takes VR to the next level. By combining our proprietary depth algorithm and multi-channel, real-time LiDAR technology, we are able to record ultra high resolution dense 3D depth information at any distance up to 100 meters, which enables our 3D capture system to perform well in any environment, indoors or outdoors."
The HDL-32E 3D sensor has proven to be an ideal fit for live action VR. Velodyne–which has made its mark in automotive, mapping, security/surveillance, and robotics–is now expanding into entertainment and film, a field that is ideally suited to what its LiDAR sensors can achieve. Being able to combine RGB colors from stereoscopic camera technology with the dense 3D point cloud from the HDL-32E for photorealistic virtual reality effects is a genuine achievement. While stereoscopic camera technology is good for up to 40-50 meters to estimate distances correctly, LiDAR adds highly accurate distance measurements with a full 360 surround view in real-time of up to 100 meters.
For cinematic VR, this ultra highresolution capture enables HypeVR to work with the highest quality digital image possible, while HypeVR’s integration of LiDAR allows a massive 3D model capture of the live action scene. Velodyne’s real-time 3D LiDAR can record depth data at a range of 100 meters with remarkable accuracy, which permits HypeVR’s virtual reality capture system to operate in any setting. Because the HDL-32E is Class 1 laser eye safe, it poses no risk of eye damage to those being filmed. While most VR rigs work best when in a fixed spot due to stitching challenges, HypeVR’s LiDAR integration allows for movement of the rig itself due to stitching being executed in 3D space.
"Some think that our first flagship VR rig, with 14 Red Dragon cameras, is overkill," Tran says. Each camera captures 6K pixels horizontally–vertically two to one, 6K by 3K pixels of data. "Yes, it’s an insane amount of data, but in order for live-action VR to truly be convincing, image quality needs to be pushed to the max. You might think that capturing an image so large is excessive for current headsets like Oculus and HTC Vive, but we don’t see it that way. The images we’re capturing are future-proof for years. And no one disputes that display technology is always going to improve."
Tran stresses that the focus needs to be on software, not on gear. "The hardware really is secondary," he says. "Creating the 360-degree rig wasn’t the real challenge. Software is the challenge, and the value. We will continue to evolve it to work faster and produce quality results with different variations of camera rigs. Initially, we’re starting at the top and will, in time, scale down to reach more market segments."
In that vein, Velodyne’s LiDAR is only a step away from extending its reach to 3D gaming and computer graphics. And because its patent-pending capture method is camera-agnostic, HypeVR is at work on more compact and affordable systems, as Tran says, other VR filmmakers "can begin creating amazing virtual reality content."
Live action virtual reality is most definitely ready for its close-up.
Wolfgang Juchmann, Ph.D., is Director of Sales & Marketing for Velodyne LiDAR, in Morgan Hill, Calif. Juchmann has more than 15 years of international experience in technical sales, product management, and marketing of industrial lasers and optical products for a variety of applications.
A 1.499Mb PDF of this article as it appeared in the magazine complete with images is available by clicking HERE