Certainty 3D LLC has developed TopoCloud, a user-friendly, practical and inexpensive tool for managing, controlling and archiving large LiDAR data sets. TopoCloud is elegantly integrated within Certainty 3Ds TopoDOT point cloud processing workflow. TopoCloud allows our customers to store their project data on a low-cost FTP/Apache server through a single administration access point. The status, updates and metadata associated with any project is easily controlled by the administrator. The administrator can assign access permission to any users worldwide. Users can access the project data and efficiently download the data they need, when they need it and then exploit a seamless integration with TopoDOT to extract the necessary information as measurements, features and CAD models.
The Problem
Prior to presenting further TopoCloud information, it is useful to consider the problem TopoCloud is designed to solve. The effective management of very large data sets has long been a challenge to the LiDAR community. Todays high speed systems offer point clouds of hundreds of millions to billions of individual points, high resolution images and additional metadata. The result are project files sizes swelling well over 100 gigabytes for a typical mobile LiDAR project. The transfer of such large files is impractical using existing internet bandwidth. Thus despite ubiquitous internet connectivity, these large files are still typically transferred via hard drive and Fedex.
The transfer problem is compounded when data is sent to more than one site as maintaining data integrity becomes a real issue. Consider four or five hard drives being sent to various separate offices and the challenges become clear. Where is the true data located? If a data modification is made, how is that communicated throughout all sites? How can the hard drives be tracked? Thus the potential for LiDAR data adding great efficiencies across downstream design, engineering and construction operations is widely acknowledged yet rarely realized given the current inefficiencies of data transfer and management.
The TopoCloud Solution
In solving the data management problem, Certainty 3D developed a list of objectives that would be incorporated into the design. We understood that much of our target market preferred or were required to store their data on-site. We decided to use a simple inexpensive storage medium owned by our customer. As we surveyed the market, we found several attempted solutionsalso in various stages of developmentwhich had incorporated very sophisticated and expensive streaming processing. We did not see that as necessary or desirable given its expense and complexity. Finally, and most importantly, we were sure to maintain the project data integrity as survey data. Thus the project data storage should be open to metadata such as control network data and documentation, CAD files, calibrated images, and any other documents necessary to the project. Above all the lineage of each LiDAR data component to the data producer, the corresponding survey control and between each of the data components must be maintained.
This resulted in the TopoCloud solution illustrated here.
An administrator will organize the LiDAR project data within TopoDOT, typically dividing up the point cloud data into tiles. The administrator will then user TopoCloud to upload the tiled data, calibrated images, survey data, documents and CAD drawings to his own FTP/Apache server. The administrator will then identify users and assign access permission for individual project(s), to each user.
Each user will be able to exploit TopoCloud integration with TopoDOT to access data through a simple graphic interface such as that shown below. Basically they will query TopoDOT and select project(s) to which they have access. The user will quickly download geospatially located icons serving as direct links to the project data in the corresponding tile located on the FTP server. One click on the world icons will download the corresponding tile from the FTP site one time after which it will be cached locally for quick access.
Admittedly downloading individual tiles may take several minutes each. So in our experience, a mile of corridor data might require 10-20 minutes to download. However, once cached locally, the data will be available in seconds. And since it typically takes a few hours to a few days to extract the desired feature and/or 3D model, the download time is insignificant.
Maintaining the data status as survey data, there is never any need nor should there be permission, for a user to stream data back. Data just flows from the FTP to the users with only the administrator allowed to upload to the FTP to change the project data.
The TopoCloud implementation allows the owner to store data locally on their FTP server as is often required. Obviously with the cost of the FTP/Apache server being about $2000-$3000, TopoCloud is a very low cost solution.
Finally TopoCloud costs are kept low by avoiding any complex streaming and/or processing operations at the server level. The information about the project, user access and data location are all maintained in parallel on the Certainty 3D license website.
TopoCloud is in the beta phase as its being tested in the field. Initial results are very promising. Certainty 3D projects TopoCloud availability to the entire TopoDOT user community by mid- to late summer 2014.