3D AI in the Lidar HD Production Process

How AI improves the point-cloud classification process at IGN, France’s national mapping agency

The Lidar HD program

Designed to meet the needs of various public actions (flood risk prevention, estimation of forest resources, etc.), the Lidar HD program aims to provide a 3D description of the French territory by 2025, using high-density aerial lidar (10 pulses/m²). Operated by the French Mapping Agency (Institut national de l’information géographique et forestière, IGN) with both internal resources and subcontracted assistance, the project is funded by governmental allocations and regional authorities’ support. The data acquired and produced as part of this program is made available as open data (point clouds, DEMs, DSMs, etc.).

The program consists of four phases: data acquisition, processing, hosting and distribution, and user support. In this article, we focus on the data-processing phase, specifically on point-cloud classification. Point clouds are segmented into ten classes: ground, low, medium, and high vegetation (categorized based on height above ground), buildings, water, bridges, permanent structures (including wind turbines, antennas, high-voltage lines, etc.), artifacts, and unclassified (Figure 1).

The area to be processed spans 550,000 km², encompassing mainland France and its overseas territories. This amounts to a massive 3 petabytes of lidar data. The data acquisition and processing are sequenced in blocks of 50 km x 50 km and must be adapted to a range of sensors (Leica Geosystems, Riegl, and Teledyne Geospatial), acquisition seasons (leaf-on or leaf-off), and diverse landscapes (plains, mountains, and overseas territories).

Tools and data for the classification process

Several tools and data sources can be leveraged for point-cloud classification:

• TerraScan¹, a tool from the TerraSolid suite, offers classification routines for classifying the point cloud using explicit instructions.
• Odeon² is a deep-learning library developed at IGN and used to create CoSIA³, which uses artificial intelligence to produce a large-scale nationwide land cover map⁴, called OCS GE (OCcupation du Sol à Grande Échelle). Based on aerial imagery, it provides a partition of the territory into 17 classes: deciduous and coniferous trees, crops, lawns, bare soil, water surfaces, greenhouses, etc.
• Myria3D⁵ is a deep-learning library designed for multi-class semantic segmentation of large-scale, high-density aerial lidar point clouds. Developed at IGN, it benefited from close collaboration with research teams highly engaged in deep-learning methods for 3D classification. Models trained in Myria3D can predict the probability of each point in a 3D cloud belonging to the classes described above.
• BD Topo® is a 3D vector description of the French territory and its infrastructures (buildings, roads, etc.), with metric precision.

The classification process combines these different sources and algorithms, resulting in improved performance compared to relying on a single method.

Two use cases for 3D AI in the process

The classification process is characterized by a sequence of steps (each represented by a TerraScan routine) that must be followed in a defined order (Figure 2). In this section, we explore two examples of the use of 3D AI in the process—buildings and ground.

Building: AI as a validator

The main idea: striking a balance between automated and manual correction

The detection of buildings is clearly crucial. Although explicit routines are used to detect buildings, they often result in over-detection (e.g. trucks, caravans, wood piles adjacent to buildings). Some sort of checking is essential.

If a group of points classified as buildings is located in an area sufficiently covered by buildings in the BD Topo, these points can be excluded from the inspection. Nevertheless, subsequent human verification remains time-consuming, sometimes trivial, and therefore tedious.

To develop a solution, therefore, the production process has been taken into consideration and a module built to be placed between the automatic detection of buildings and their verification by an operator, with the goal of reducing the number of verifications.

This module is tasked with differentiating between groups of points identified as buildings:

• Those that do not require inspection because the AI, in agreement with BD Topo, is confident that they are buildings
• Those that do not require inspection because the AI, in accordance with BD Topo, is confident that they are not buildings
• And those for which the AI cannot make a confident determination.

Only in the case of the third bullet above will the operator be asked to make a verification (Figure 3).

Training data: buildings only

In AI, training data is a crucial resource: its availability and quality directly impact model performance. The dataset used to train the first AI model covers various urban and rural landscapes, spanning an area of 150 km², but features a single class: buildings. The points’ labels have been verified and curated by operators. As a result, the trained model can predict with considerable accuracy whether or not a point should be classified as a building.

Procedure: optimization under constraint of decision thresholds

The process works on points that are grouped into connected components by proximity (Figure 4). For each component, a decision is made to classify it as a building, not a building, or an uncertain case. This decision is based on two sources of truth:

• The point cloud processed by the AI model, giving a building probability for each point.
• Building vectors from BD Topo.

The decision is also based on thresholds: a point is validated if its probability is sufficiently high, and refuted if its probability is sufficiently low, and when the proportion of validated (or refuted) points of a component is sufficiently high, that component is confirmed (or refuted) according to each of the sources. These decisions are then cross-referenced to apply the final decision to all the points in the connected component (Figure 5).

The five decision thresholds used in this process are optimized to maximize automation, i.e. to reduce the number of groups of points to be verified, under the constraint of maintaining fixed quality performance (precision and recall above 98%) (Figure 6).

Results

There is a synergy between the two sources due to the joint optimization of thresholds. AI compensates for the shortcomings of BD Topo, and BD Topo makes up for the erroneous refutations of the AI model.

By January 2022, 85% of point groups could be verified automatically. Since then, continuous improvement of the AI model (multiclass model, variety of training data) has enabled decisions for 96% of point groups to be automated, resulting in a significant reduction in the visual inspection time required to process these uncertainties.

Ground: using AI to fill in the gaps

Classification of points as ground is of major importance too and presents problems of its own. Several of these have been addressed and effective solutions developed.

Ground detection on ridges

Explicit routines for ground detection often struggle to detect ridges. These routines work by densifying low points via triangulation, where ground slopes are limited to avoid classifying buildings as ground. However, this approach often fails to detect rocky ridges. As a result, the bare-earth digital terrain model (DEM) can have unsightly triangulations (Figure 7a). The question is whether the 3D AI model can detect ground on these ridges.

Training data: 151 km² multiclass

The model is a multiclass one, trained on a 151 km² dataset spanning three administrative departments. The classification of this area is the result of an automatic process followed by manual corrections. These 151 km² do not include any mountainous areas, but they do include a few square kilometers covering part of the Gorges du Tarn, a canyon several hundred meters deep carved out of limestone by the Tarn River.

Process: bringing out the best in everyone

The results of the AI model are not perfect either: ridges are well detected, but there are artifacts that resemble small rocks (Figure 7b).

Surprisingly, the two approaches are complementary: the AI results are good in areas where the explicit routines perform poorly, and vice versa. Therefore, the decision was made to combine the two sets of results. In cases where the explicit routine yields a low density of ground points, the points classified as ground by the AI model are used instead.

Results: astonishing complementarity

The result (Figure 7c) is surprising: the two sets of data complement each other perfectly, even in the Alps, a mountain range with vertical drops that are much greater than those of the Gorges du Tarn!

Future perspectives: greenhouses, hydrography, vegetation…

Other classes are likely to benefit from the complementarity of the different data sources. Tests are currently being finalized and prepared for deployment to process greenhouses, hydrography, and vegetation using explicit routines based on predictions from Odeon models (2D AI), and Myria3D models (3D AI).

Summary and perspectives

The use of Myria3D, an AI framework designed for 3D processing, has significantly improved results. The use cases described in the previous paragraphs have been in production for over a year.

From the beginning, the decision was made to position Myria3D AI as a supplementary tool to augment other methods (explicit routines) and data sources (BD Topo, OCSGE probability map), leveraging the Institut’s expertise in lidar (creation of a cross-disciplinary AI team) and the use of the TerraScan tool to coordinate the various sources.

AI was gradually incorporated into the classification process, according to improvement priorities (buildings, then ground; hydrography, vegetation, and greenhouses to follow), available data, and results (the subject of bridges was not pursued, as the tests conducted did not yield conclusive results).

To contribute to the 3D AI research ecosystem, we have made the Myria3D code open source⁶, and the training dataset for the latest AI model will also be made public soon. Lidar HD data is openly available⁷.

Several perspectives are currently being examined.

• As the Lidar HD program progresses, more and more data becomes available to train our 3D AI models. From an initial situation of scarcity, we now have access to abundant training data. We therefore developed methods to consolidate a highly diverse training dataset via the sampling of 60,000 50 m x 50 m data patches spanning 150 km². The Pacasam tool⁸, developed in-house and open source, contributes to this effort.
• The current architecture of the model is based on research conducted in 2021. Using more recent model architecture, such as Super Point Transformer⁹, may lead us to more energy-efficient AI models with competitive results.
• Experiments have been conducted using Myria3D’s architecture to distinguish between different forest species.

Floryne Roche graduated with a degree in geomatics engineering from Ecole national des Sciences Géographiques, in Marne-La-Vallée, near Paris, and began her career at the Institut national de l’Information géographique et forestière (IGN), where she developed tools for processing altimetry, particularly from lidar data. For seven years, she served as the technical link between the users of IGN products and services and the Institut, gaining a deep understanding of the business needs of her contacts. Currently, as Product Owner of the AI team within the Lidar HD program, she ensures that AI is effectively integrated into the production process, while also working to promote a clear understanding of its contributions.

---