Integrate sparse convolutions to autoware

[knzo25]

Background

In 3d object detection, so far we have constrained ourselves to the family of models that use traditional operations (convolutions, dense layers, attention mechanism, etc). That is one of the main reasons why we use centerpoint and transfusion with pillar based backbones.

In the ever evolving SOTA, there are many models that do not use sparse convolutions, but a great portion of them do, specially those that are focused on deployment (memory consumption, inference speeds, etc)

A quick glance at the code of the top ranking methods in object detection and segmentation will provide some insight on how popular sparse convolutions are:

• https://paperswithcode.com/sota/3d-semantic-segmentation-on-semantickitti
• https://paperswithcode.com/sota/3d-object-detection-on-nuscenes
• https://paperswithcode.com/sota/3d-object-detection-on-waymo-open-dataset

I would like to propose integrating the most popular sparse convolutions backbone into autoware (i.e, traveller59's backend).
Some alternatives are:

• traveller59's backend:
  • Apache 2.0
  • Fully open source
  • Standard in ML (fastest, low memory consumption).
  • Deployment aware: static allocation suited for inference engines (i.e., tensorrt)
  • Directly compatible with most models out there ! (by default it can not be exported to ONNX nor executed by TensorRT, but we have addressed those limitations in this repository)
• MMCV's implementation:
  • Apache 2.0
  • Fully open source
  • traveller59's implementation is preferred in popular repositories like mmdet3d
  • Forces us to use LibTorch (autoware got rid of this dependency in the past)
  • Not very suited for inference
• Lidar AI Solution's implementation:
  • MIT license
  • Closed source shared library
  • Developed by NVIDIA, so its performance should be high

Proposals

traveller59's implementation is a python package that generates c++/cuda code.
To integrate it into autoware, I see the following alternatives:

• Integrate it into autoware's setup script fully:
  • Add lines like the following to the setup script.
  • Probably requires torch as a python library to generate the code
  • Outsource the responsibility to the upstream repository
  • Users will only need to build it once. Takes up to 5 minutes in a desktop environment (using all the cores)
• Integrate a custom repository to the setup script
  • Generate the c++/cuda code in a separate environment
  • Create a repository with the generated code
  • Build the repository as a part of the setup script
  • We can keep control of the build process and the maintenance. For every major cuda release, we probably will need to regenerate the code (takes around a few minutes)
  • Users will only need to build it once. Takes up to 5 minutes in a desktop environment (using all the cores)
• Integrate it as a ros2 package and add it to the repos file:
  • Generate the c++/cuda code in a separate environment
  • Create a repository with the generated code. Wrap it around a ros2 package
  • Build it as a normal ros dependency
  • We can keep control of the build process and the maintenance. For every major cuda release, we probably will need to regenerate the code (takes around a few minutes)

TODO List

• Define if and how to integrate the library (if there is another way to do this, please let me know 🙏 )
• Implement the agreed upon idea

[liuXinGangChina]

Great proposal，kenzo san @knzo25， looking forward to your master piece！

By the way，We have a group discussion during Software working group meeting this week , and get a conclusion that we plan to integrate bevfusion nvidia optimized version（which is NVIDIA-AI-IOT） into autoware，（now we are working on nodes migration from autowre universe to core， hopely integration will start late January）

Have a nice day！

心刚

[knzo25]

Thanks for the update. I heard from some of the participants that we will have both implementations so users can choose according to their needs (no need to be restrictive in OSS as long as maintainers are active).
Maybe we can help each other during the review process so we can integrate the packages sooner

[cyn-liu]

Great proposal, @knzo25 , looking forward to your next step of work!

I prefer to support the third method.

Integrate it as a ros2 package and add it to the repos file:
Generate the c++/cuda code in a separate environment
Create a repository with the generated code. Wrap it around a ros2 package
Build it as a normal ros dependency
We can keep control of the build process and the maintenance. For every major cuda release, we probably will need to regenerate the code (takes around a few minutes)

As an Autoware user, I am used to pulling code and mounting it into the docker environment for compilation, and do not pay attention to the setup script of Autoware. Therefore, the third method treats spconv as an independent ros2 package and is pulled together when pulling Autoware, which is more visible and easier to manage.

When an error occurs during compilation of package spconv, it is easier to locate the cause.

The frequency of CUDA version upgrades in Autoware's docker environment is very low, so the frequency of code regeneration is very low.

[knzo25]

Thanks for the feedback !
Here we have our own development pipeline (we have not all yet migrated to docker environments) and was worried about how people from the community would prefer the integration

[ahmedius2]

Hello,

I integrated OpenPCDet into autoware, trained PillarNet (which utilizes spconv), tested it in AWSIM, and it works well. I could help you with the process but it appears to me that the lidar object detection models are moving away from sparse convolution.

[knzo25]

@ahmedius2
Thanks! Can you provide details what you mean by integrating OpenPCDet into autoware?
Our usual flow so far is to train models into mmdet3d/openpcdet, generate an onnx, create a ROS node that generates an engine from the onnx, and run preprocess/inference/postprocess in the ROS node. Did you make a generic wrapper for OpenPCDet in ROS? or a python node that runs inference directly using OpenPCDet?

On another node, if we end up accepting a ROS wrapper for spconv, I made a proposal here
https://github.com/knzo25/ros2_spconv

[ahmedius2]

It is a ROS2 python node that runs inference directly on OpenPCDet. When running the DNN, it utilizes TensorRT to accelerate the inference for dense convolutions but it executes spconv and prior parts as it is in Python. It also utilizes torchscript in various parts. In addition, I implemented a ROS2 node that merges multiple consecutive point clouds into one while considering the vehicle's ego-pose that corresponds to each cloud's sampling time. This is how it is done in OpenPCDet when models are trained/tested with multiple sweeps to increase their accuracy and allow velocity prediction.