Point Cloud Segmentation Using Dynamic Graph CNNs
In this article, we explore a simple point cloud segmentation pipeline using Dynamic Graph CNNs, implemented using PyTorch Geometric along with Weights & Biases.
Created on December 22|Last edited on May 24
Comment
Point clouds are an unordered set of 2D or 3D points in space that provide a flexible geometric representation suitable for countless applications in computer graphics and vision. With the advent of fast 3D point cloud acquisition, recent pipelines for graphics and vision often process point clouds directly, bypassing expensive mesh reconstruction or denoising due to efficiency considerations or instability of these techniques in the presence of noise.
However, such modern applications demand high-level processing of point clouds, which have long been tackled using hand-designed features and algorithms. Rather than identifying salient geometric features like corners and edges, these algorithms search for semantic cues and affordances. These features mostly do not fit cleanly into the frameworks of computational or differential geometry. They typically require learning-based approaches that derive relevant information through statistical analysis of labeled or unlabeled datasets.
In this article, we will explore the paper Dynamic Graph CNN for Learning on Point Clouds. Notably, this paper proposes a CNN-based approach for machine learning on 3D point cloud data. It's worth noting that point clouds inherently lack topological information, so designing a model to recover topology can enrich the representation power of point clouds.
To this end, we'll explore a new neural network module dubbed Edge Convolution, proposed by the aforementioned paper, one that's suitable for CNN-based high-level tasks on point clouds, including classification and segmentation.
Here's what we'll be covering:
Table of Contents
Diving Into the WorkflowThe ShapeNet DatasetDelving Into Dynamic Graph CNNsExisting Approaches for Ingesting Point Cloud DataHow Are These Drawbacks Resolved?Training Point Cloud Segmentation ModelExperimentsConclusion
Diving Into the Workflow
In this article, we'll be using PyTorch Geometric to create a deep-learning pipeline for implementing and training our model for 3D point cloud segmentation.
PyTorch Geometric, also referred to as PyG, is a library built upon PyTorch to easily write and train Graph Neural Networks and Geometric Deep Learning models for a wide range of applications related to both structured and unstructured data. PyG consists of the following features:
- Various methods for deep learning on graphs and other irregular structures from a variety of published papers.
- A large number of common benchmark datasets (based on simple interfaces to create our own datasets).
- Easy-to-use mini-batch loaders for operating on many small and single giant graphs and other irregular data structures such as molecules, 3D meshes, and point clouds.
We would also be using Weights & Biases for experiment tracking, data visualization, model checkpoint management, and hyperparameter tuning.
The ShapeNet Dataset
We'll be demonstrating our point-cloud segmentation model by training and evaluating our model on the ShapeNet Dataset, which consists of 16 categories of point clouds of 3D CAD models. To explore the complete dataset interactively, know more about its usage in a machine learning workflow for point cloud data and check how different models are performing on this dataset, you can refer to the following article.
Delving Into Dynamic Graph CNNs
Adapting deep learning to point cloud data is not very straightforward. Standard deep neural network models require input data with a regular structure, while point clouds are fundamentally irregular: the positions of the points are continuously distributed in the space, and any permutation of their ordering does not change the spatial distribution (permutation-invariance).
Existing Approaches for Ingesting Point Cloud Data
- One common approach to processing point cloud data using deep learning models is to first convert raw point cloud data into a volumetric representation, namely a 3D grid. This approach, however, usually introduces quantization artifacts and excessive memory usage, making it difficult to go to capture high-resolution or fine-grained features.
- Some deep neural networks are designed specifically to handle the irregularity of point clouds, directly manipulating raw point cloud data rather than passing to an intermediate regular representation.
- PointNet achieves permutation invariance of points by operating on each point independently and subsequently applying a symmetric function to accumulate features.
- PointNet++, which is an extension of PointNet, considers neighborhoods of points rather than acting on each independently. This allows the network to exploit local features, improving upon the performance of the basic model.
- These approaches, although they respect the permutation-invariance of the point cloud data, are unable to model the geometric relationships among points, presenting a fundamental limitation that cannot capture local features.
How Are These Drawbacks Resolved?
Key contributions of the paper Dynamic Graph CNN for Learning on Point Clouds
0
Training Point Cloud Segmentation Model
Next, let's get training. Specifically, we'll:
- Train our model using a simple PyTorch-based training loop.
- Use negative log-likelihood as our loss function.
- Use accuracy and intersection-over-union (or IoU) as our evaluation metrics.
- Leverage Weights & Biases for tracking our experiment metrics, versioning model checkpoints, and creating detailed visualizations. For detailed documentation of writing PyTorch-based training loops instrumented with Weights & Biases, check out the official documentation.
Run set
0
Experiments
Now that we have our input pipeline, architecture, and training loop implemented, let's train a few models for point cloud segmentation. Here, we'll use some of the categories of point clouds available as part of the ShapeNetCore subset of the ShapeNet Dataset. We'll also evaluate our model post-training on the test split of our dataset.
Training and Validation results for Airplane
2
Training and Validation results for Car
2
Training and Validation results for Rocket
2
Training and Validation results for Table
2
Training and Validation results for Chair
2
Conclusion
In this article, we explore how to build a model for segmenting 3D point clouds.
- We explored a Dynamic Graph CNN model for segmenting 3D point clouds as proposed by the paper Dynamic Graph CNN for Learning on Point Clouds.
- We explore the EdgeConv module as the building block of our model and how we can build it easily using PyTorch Geometric.
- We trained and evaluated our model on different categories of the ShapeNetCore dataset. In order to explore this dataset interactively and check the performance of various models on this dataset on a leaderboard, check out the following article:
- In order to check how to build a model for the classification of point clouds, check out the following report:
Add a comment
Iterate on AI agents and models faster. Try Weights & Biases today.