Overview

3D perception of dynamic, deformable objects is a fundamental problem that has broad applications in industrial, household, and surgical settings. Deformable objects can take an infinite variety of configurations that make it difficult to perceive their semantically meaningful parts. Geometric deep learning methods have proven effective in handling local geometric structures within rigid point clouds. However, they struggle when faced with highly deformable objects undergoing continuous spatial transformations that alter the Euclidean distances and angles between object parts. Topological data analysis (TDA) can model the global connectivity structure of an object, which remains stable even for highly deformable objects. Therefore, we propose to improve 3D perception of deformable objects by fusing topological and geometric information. We leverage the Mapper algorithm, a tool from TDA for modeling global connectivity by representing the topological structure of data as a graph. We extend the Mapper algorithm to operate on time-varying deformations with the Sequential Mapper algorithm. We then propose the TopoMetric Graph Neural Network (TOM GNN) for learning topological and geometric features to efficiently and accurately perform semantic part segmentation. We benchmark this technique for 3D semantic part segmentation on our novel dataset called FabricDeform3D, which includes 75,000 labelled point clouds with 65 semantic part labels across 15 highly deformable object classes of garments, bags, and backpacks. Compared to well-known point cloud processing backbones, TOM GNN outperforms the strongest baseline by 1.9% with the mean IoU metric on FabricDeform3D. TOM GNN also generalizes to static rigid point clouds on the standard benchmark ShapeNet-Part.