A 1.406Mb PDF of this article as it appeared in the magazine complete with images is available by clicking HERE
Smart phones are becoming smarter everyday by integration of the latest sensor technologies with these user friendly handheld devices. Mobile augmented reality (AR) applications utilizing geospatial data are being used to provide various location based services to mobile users in real time. Navigation using mobile based AR services will become more popular when a user can view a virtual environment on his/her mobile device. Thinking apart from navigation services a user may also be interested in more details about his surroundings, i.e., buildings, electric or telephone poles, trees, sign boards, or landmarks.
Thinking Beyond
Current apps make it possible to identify features in one’s surrounding by their name and location using popular mobile map based services. A mobile user will appreciate such services more if he/she is also able to determine the dimensions of the features onsite in real time. However, such an AR service does not exist to the best knowledge of the authors.
In view of this, the objective of this research work is to make it possible for a mobile user to obtain dimensions of buildings and other features on his/her mobile device by just clicking over the live photograph of the scene. This will have wider applications to a variety of users, e.g., a field engineer, a tourist, or a municipal officer by facilitating dimensional information about one’s surroundings.
Proposed Methodology
The basis of our approach is to integrate a photograph taken on a mobile device (with which a user interacts in field) with the point cloud data that is stored on a server. Through this integration, dimensions are generated using the point cloud for the desired locations as communicated by the mobile device to the server, and thereby display these dimensions onto the photograph on the mobile.
The point cloud data can be from any source including and mainly LiDAR data, which contains coordinates and intensity information of a large number of points. Dimensions can be easily computed between any two points using LiDAR data. LiDAR data of buildings and surroundings provide information regarding geometry of various features. As discussed, the need is to get the dimensions of the building or any other feature in the captured mobile image using LiDAR point cloud database of the same scene. This will require 2D to 3D registration of the datasets between mobile RGB image and intensity image of LiDAR data. A user in the field can have different viewpoints of the landscape and getting the corresponding 3D data is challenging. The results obtained so far validate the feasibility of the proposed framework and algorithm.
A mobile device is usually equipped with a camera, GNSS and gyroscope hardware which provide the location and orientation parameters of mobile images. Our proposed method attempts to generate a pseudo image using 3D point cloud data corresponding to a live mobile image (Figure 1). This pseudo image is generated by projecting the point cloud corresponding to the live image onto a plane defined by the live mobile image and using the intensity information of point cloud as the digital numbers of this pseudo image. The external and internal orientation (EO/IO) parameters of the live mobile image are utilized for this purpose along with the collinearity equations. The pseudo image thus generated may have differences in illumination, image resolution, viewpoint and scale owing to uncertainties in EO/IO parameters. Having generated such pseudo image the next step is to find correspondences between the live mobile image and pseudo image using scale and rotation invariant registration method.
Extensive literature is available on various image registration methods, but the method required in the proposed framework demands a robust registration technique. To deal with the issue of registering a good resolution image with a less reliable intensity (pseudo) image, we selected the feature based matching approach to detect features and their descriptors in both the images so that descriptors can be used as a basis to find correspondences between the two images.
For our work we chose Lowe’s SIFT algorithm which is a suitable choice for scale and rotation invariant feature detection and matching. The selected matching approach was found successful in realizing matching between live and pseudo image as in our case. At this point, a few successful matches in both the images were used to determine transformation parameters to transform one image to the other using homography. Thus, 2D live mobile image is registered to intensity (pseudo) image obtained using 3D LiDAR data to provide point to point correspondence between these images. This facilitates locating a point in 3D point cloud corresponding to a point in the live mobile image. Similarly, corresponding to a pair of points on the live image the distance between these is computed using the 3D point cloud and augmented over the live image. Full technical details of our algorithm are being published elsewhere.
Experiments and results
To validate the proposed approach we acquired LiDAR data of an area in our campus using ILRIS 3D Terrestrial Laser Scanner (TLS). The geo-referenced point cloud obtained covers all buildings and other features in the chosen area. To simulate the mobile phone data capture we used a camera, GPS and IMU combination therefore knowing the EO parameters of the camera. The whole experimental setup is shown in Figure 2. A couple of images were captured in field using this setup.
Using the proposed approach the camera images are successfully registered to the resulting intensity (pseudo) image of LiDAR data. One such example is shown in Figure 3. Dimensions corresponding to a few point pairs on the photograph were then obtained and augmented on the photograph as shown in Figure. 4.
Future scope
The results obtained show the feasibility of the proposed methodology to extract dimensions and augment these over a mobile image onsite. The experiment conducted shows that the dimensions computed are correct for approximately 80% of the total queries made. The prototype realized so far is to demonstrate the concept and its applicability. The future work will involve implementing the system using a mobile device and a server, as proposed originally. We are also working on the methods to improve accuracy of dimension computation and to make the process more robust and reliable. This system has the potential to benefit mobile users who need dimensions of a site. It is assumed here that point cloud data are available in a server. However, the same may also be replaced by other geospatial data, e.g., CityGML representation of terrain.
Dr. Bharat Lohani is Associate Professor at IIT Kanpur and focuses his research in LiDAR technology development and applications.
Surabhi Gupta is a master’s student at IIT Kanpur in geoinformatics with research focus on augmented reality applications using geospatial domain.
A 1.406Mb PDF of this article as it appeared in the magazine complete with images is available by clicking HERE