A 1.482Mb PDF of this article as it appeared in the magazine complete with images is available by clicking HERE
The volume of 3D point cloud data from LiDAR, multi-beam echo sounding, and dense image matching from photos is rapidly increasing as technological advancements continue to make acquisition cheaper and easier. Point cloud data are complex (multi-dimensional), are large in size (million to billions of points) and require special handling for efficient access and analysis. The US government is one of the largest users and producers of these data, and in particular across the Department of Defense (DoD), new sensor technologies such as Geiger mode LiDAR are quickly becoming the standard for operational use.
The Geospatial Repository and Data (aka GRiD) Management System is a storage and management solution that meets the challenge of complex archive, management, and dissemination of these data by allowing users to interrogate, filter, subset, and process point cloud data and derivative products. It is experiencing significant growth in total point cloud holdings stored, geographic area coverage, and number of active users (currently 2500+) as users find more and varied uses for the unique data it manages.
This article will describe some of the capabilities of the GRiD system, describe its somewhat unique-to-government development methodology, provide a status update of the current holdings, and inform potential federal government users and their partners where they can connect to the system for their point cloud access and processing needs.
Background
Initially developed as a research prototype, GRiD was deployed as an interactive spatial database capability for the storage and management of DoD LiDAR point cloud holdings and derivative products in the Fall of 2012. The GRiD effort leverages development methodologies from open source software to accelerate development, reduce sustainment costs, encourage community collaboration, and ensure consistency in data format specifications and standards. Numerous open source point cloud software solutions were originated or modified in support of GRiD development activities.
The GRiD concept, initial and ongoing research and development and operational support, is provided by the US Army Corps of Engineers (USACE) Cold Regions Research and Engineering Laboratory (CRREL), in Hanover, New Hampshire, in collaboration with other federal agencies.
While GRiD’s primary development focus has been DoD users, its technology and capabilities are very useful in civilian contexts as well. USACE manages a civilian GRiD instance to support federal government users, and federal employees can sign up to get access to computing, processing, and visualization resources of the system.
Capabilities
GRiD manages point cloud data for users in a way that works to protect them from its often-overwhelming volume. Data are processed, organized, and compressed using the open source Point Data Abstraction Library (PDAL) before ingestion into an Oracle database at the heart of GRiD. A website allows users to quickly subset, perform ad-hoc queries, and apply exploitation algorithms, and filter data by sensor, attributes or collection date.
While wide-area mapping efforts capture large swathes of base information, many GRiD users often only need data for focused areas. GRiD optimizes for that scenario by implementing a spatial indexing technique that balances the storage of the point cloud data with increased selectivity for small areas. Data are normalized into a typical database schema that allows for interrogation of collection metadata, and users can pivot queries based on location, collection time, and acquisition sensor. These features allow users to quickly drill down to a subset focused to meet their needs.
In addition to subset and interrogation, users are also able to execute exploitation algorithms on their data. Bare earth extraction and classification, digital terrain model or surface model generation, elevation model-derived products such as slope, line-of-sight analysis, helicopter landing zones, and coordinate system reprojection are just some of the possible actions users can apply after making a data selection. These kinds of server-side processing activities allow bandwidthand processing-disadvantaged users to accomplish sophisticated analysis in limited environments.
Development Methodology
GRiD utilizes, supports, and contributes to open source software which enables it to accomplish its mission. The primary contribution has been in the support of the PDAL software project. GRiD uses PDAL for data integration, processing, format translation, and management. Software and algorithms are built as PDAL components which can then be seamlessly integrated into a larger system.
GRiD development activities have also resulted in support of projects such as LASzip, libLAS, GDAL, and MapServer. Like many server-side technologies, GRiD is built upon a wide stack of open source software, and the project contributes new features and capabilities back to the projects on which it depends. Because of open source software, GRiD is able to proudly stand on the shoulders of those technologies that have made it possible. Likewise, other efforts looking to integrate with GRiD are able to easily operate at the software level using components with no administrative impediments or licensing restrictions on software.
Status Update
GRiD systems currently manage over 1.9 million total files (point cloud, imagery and elevation models) totaling over 250 TB of data including nearly 5.5 trillion LiDAR points. Users may access GRiD using the website interface and through Open Geospatial Consortium services such as Web Mapping and Web Feature Services. GRiD is developing an application programmer’s interface called Greyhound that will formalize access to point cloud holdings for external developers and is participating in the nascent Pointdown effort to construct an API specification for progressive point cloud access.
GRiD is also working to integrate visualization capabilities in two ways. First, it is collaborating with Applied Imagery to enable Quick Terrain Modeler users to effortlessly interrogate and fetch point cloud and terrain data from GRiD. Second, it is supporting the open source development of http://plas.io, which is a WebGL environment for visualization of point cloud data in web browsers. GRiD uses plas.io to allow users to quickly preview data inside their web browsers before downloading it.
Find out more
Visit https://lidar.io for more information about GRiD. Federal employees and military personnel can easily obtain access by signing up online or by contacting GRiD.Core.Team@erdc.dren.mil. Contact GRiD to act as a dissemination and access mechanism if your organization is planning an acquisition or has data that is not easily distributed.
People curious about the software should visit PDAL and plas.io to find out how these open source technologies could be used in your own processing and workflow chains. PDAL software is available as both Windows executables and Linux source distributions, and plas.io is available as Javascript you can embed in your own web pages.
Conclusion
The ongoing success of LiDAR, the adoption of technology for conversion of other sources to point cloud data, and the increased awareness that 3D data provides will continue to propel interest in the collection and storage of point cloud data. GRiD allows users to exploit and interact with point cloud data in their web browsers, and the project has spun off numerous open source technologies which can be leveraged in other contexts. Federal employees should sign-up today to learn how they can use this capability in support of their own activities.
David C. Finnegan is a Research Physical Scientist with the US Army Cold Regions Research and Engineering Lab (CRREL) in Hanover, NH. David has 15+ years designing, developing and fielding advanced LiDAR sensor and dissemination technologies with academia, the DoD and the private sector.
Howard Butler leads a small company of open source software developers in Iowa City called Hobu, Inc. He and his firm have led the development of open source projects like PDAL, libLAS, plas.io, and Greyhound which work together to provide a processing and visualization stack of capabilities in the LiDAR and point cloud domain.
A 1.482Mb PDF of this article as it appeared in the magazine complete with images is available by clicking HERE