Awash in Data (Part I)

Recently I attended the SPAR Conference on laser scanning. It’s exciting to see new innovations and to talk with practitioners about their experiences, successes and challenges. I noticed a recurring theme that echoed from all corners. Many hardware and software vendors, service providers and end users indicate that we are on the brink of a data explosion, and that dealing with huge volumes of information will soon become a major focus. One expert went so far as to call the data explosion a big problem, but it is more reasonable to recognize growth, especially explosive growth, as a good thing, and to identify the challenges and opportunities that lie ahead.

With this in mind, this article is divided into two pieces: this column (Part I) examines the macro-trends underlying the proliferation of data, and the next (Part II) will explore what can be done today, as well as what users and practitioners should expect or perhaps demand from vendors (especially software) in the future.

Fundamentally, the data explosion is driven by the fact that the world is moving to 3D. For centuries, scientists, engineers, artists, cartographers and the like were forced to work with 2D representations of reality because of technical limitations. However, the advent of advanced computing and graphics has spurred the 3D movement just look at movies, TV, computer games or CAD systems. Laser scanning will follow the trend because it fulfills the need for efficient, high-quality, accurate, and realistic 3D content.

Why now? Laser scanning has become an accepted industry practice. What just a few years ago was relegated to scientists or specialty engineering is now commonplace. The word is getting out: a few weeks ago I attended a social function at a local bank. Out of the blue, our host a risk management vice-president informed us of plans to laser scan their building, which has historical significance. It was something quite gratifying to hear, especially when accepted as commonplace by someone who does not deal with this type of technology.

More importantly, organizations are beginning to mandate the use of laser scanning as a best practice. For instance the GSA has plans to document all federal buildings and will use laser scanning as part of the process: one presentation at SPAR showed how laser scanning was used to document a 1.4 million square-foot building; data storage requirements are easily in the hundreds of gigabytes.

Moreover, there is serious talk of using vehicle-based mobile mapping to document millions of miles of highways nationwide, since it is now possible to drive down a highway at a reasonable speed and in a few minutes collect gigabytes of detailed data about the road surface, markings, signage, overpasses, tunnels and other features. For such an application, scanners can achieve centimeter-level resolution: a data explosion is clearly inevitable.

The force behind the data explosion is the simple economics of supply and demand. From the supply perspective, the cost of collection has been steadily dropping. Some newer scanners can deliver over a million measurements per second practically a hundredfold improvement over earlier systems. The scanners are shrinking in size and becoming more user-friendly. In fact, scanners have gotten so fast that in many cases the actual scanning time is almost negligible compared to the time for setup and movement of equipment. So it should be no surprise that one new product unveiled at the conference is an indoor mapping system: essentially a pushcart that automatically collects and registers data while it is wheeled through a facility. Now that’s fast!

Another very recent development is the fusing of point measurements with intensity or color information. If you haven’t seen colorized point clouds yet, do so; you’re in for a treat. Adding color makes the data much more visually appealing and useful. Along the same lines, many vendors now collect still photos at the same time as the scans, allowing users to see the environments of interest in very high resolution. This additional imagery and colorization increases storage significantly. On top off all this new innovation, prices have been steadily declining, spurring more demand, which in turn will spur more innovation and even lower prices.

All this is great news, but it means that we are soon to be awash in data. In Part II, we will examine solutions for collecting large volumes of data and also for extracting higher-value information from huge datasets. Stay tuned.