nuPlan

nuPlan is a planning simulator that comes with a large-scale dataset for autonomous vehicle research. This dataset contains ~1282 hours of driving logs, including ego-vehicle data, HD maps, and auto-labeled bounding boxes, spanning 4 cities. About 120 hours of nuPlan include sensor data from 8 cameras and 5 LiDARs.

Overview

Papers

Towards learning-based planning: The nuplan benchmark for real-world autonomous driving

nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles

Download

nuplan.org

Code

nuplan-devkit

License

CC BY-NC-SA 4.0

nuPlan Terms of Use

Apache License 2.0

Available splits

nuplan_train, nuplan_val, nuplan_test, nuplan-mini_train, nuplan-mini_val, nuplan-mini_test

Available Modalities

Name

Available

Description

Ego Vehicle

✓

State of the ego vehicle, including poses, dynamic state, and vehicle parameters, see EgoStateSE3.

Map

(✓)

The HD-Maps are in 2D vector format and defined per-location. For more information, see MapAPI.

Bounding Boxes

✓

The bounding boxes are available, see BoxDetectionWrapper.

Traffic Lights

✓

Traffic lights include the status and the lane id they are associated with, see TrafficLightDetectionWrapper.

Pinhole Cameras

(✓)

Subset of nuPlan includes 8x PinholeCamera:

Fisheye Cameras

X

LiDARs

(✓)

Subset of nuPlan includes 5x LiDAR:
Dataset Specific
class py123d.conversion.registry.NuPlanBoxDetectionLabel[source]

Semantic labels for nuPlan bounding box detections.

VEHICLE = 0

Includes all four or more wheeled vehicles, as well as trailers.

BICYCLE = 1

Includes bicycles, motorcycles and tricycles.

PEDESTRIAN = 2

All types of pedestrians, incl. strollers and wheelchairs.

TRAFFIC_CONE = 3

Cones that are temporarily placed to control the flow of traffic.

BARRIER = 4

Solid barriers that can be either temporary or permanent.

CZONE_SIGN = 5

Temporary signs that indicate construction zones.

GENERIC_OBJECT = 6

Animals, debris, pushable/pullable objects, permanent poles.

to_default()[source]

Inherited, see superclass.

Return type:

DefaultBoxDetectionLabel

class py123d.conversion.registry.NuPlanLiDARIndex[source]

LiDAR Indexing Scheme for the nuPlan dataset.

X = 0
Y = 1
Z = 2
INTENSITY = 3
RING = 4

Download

You can install the nuPlan dataset either by downloading the files from the official website or by using the following bash script:

Download Scripts

License:

# NOTE: Please check the LICENSE file when downloading the nuPlan dataset
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/LICENSE

Maps (required for nuplan and nuplan-mini):

wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-maps-v1.1.zip

Logs:

# 1. nuplan_train
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_train_boston.zip
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_train_pittsburgh.zip
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_train_singapore.zip
for split in {1..6}; do
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_train_vegas_${split}.zip
done

# 2. nuplan_val
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_test.zip

# 3. nuplan_test
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_val.zip

# 4. nuplan-mini_train, nuplan-mini_val, nuplan-mini_test
wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/nuplan-v1.1_mini.zip

Sensors:

# 1. nuplan_train
for split in {0..42}; do
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/train_set/nuplan-v1.1_train_camera_${split}.zip
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/train_set/nuplan-v1.1_train_lidar_${split}.zip
done

# 2. nuplan_val
for split in {0..11}; do
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/val_set/nuplan-v1.1_val_camera_${split}.zip
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/val_set/nuplan-v1.1_val_lidar_${split}.zip
done

# 3. nuplan_test
for split in {0..11}; do
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/test_set/nuplan-v1.1_test_camera_${split}.zip
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/test_set/nuplan-v1.1_test_lidar_${split}.zip
done

# 4. nuplan_mini_train, nuplan_mini_val, nuplan_mini_test
for split in {0..8}; do
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/mini_set/nuplan-v1.1_mini_camera_${split}.zip
    wget https://motional-nuplan.s3-ap-northeast-1.amazonaws.com/public/nuplan-v1.1/sensor_blobs/mini_set/nuplan-v1.1_mini_lidar_${split}.zip
done

The 123D conversion expects the following directory structure:

$NUPLAN_DATA_ROOT
  ├── maps (or $NUPLAN_MAPS_ROOT)
  │   ├── nuplan-maps-v1.0.json
  │   ├── sg-one-north
  │   │   └── 9.17.1964
  │   │       └── map.gpkg
  │   ├── us-ma-boston
  │   │   └── 9.12.1817
  │   │       └── map.gpkg
  │   ├── us-nv-las-vegas-strip
  │   │    └── 9.15.1915
  │   │        └── map.gpkg
  │   └── us-pa-pittsburgh-hazelwood
  │       └── 9.17.1937
  │           └── map.gpkg
  └── nuplan-v1.1
      ├── splits
      │     ├── mini
      │     │    ├── 2021.05.12.22.00.38_veh-35_01008_01518.db
      │     │    ├── 2021.06.09.17.23.18_veh-38_00773_01140.db
      │     │    ├── ...
      │     │    └── 2021.10.11.08.31.07_veh-50_01750_01948.db
      │     ├── test
      │     │    └── ...
      │     └── trainval
      │          ├── 2021.05.12.22.00.38_veh-35_01008_01518.db
      │          ├── 2021.06.09.17.23.18_veh-38_00773_01140.db
      │          ├── ...
      │          └── 2021.10.11.08.31.07_veh-50_01750_01948.db
      └── sensor_blobs (or $NUPLAN_SENSOR_ROOT)
            ├── 2021.05.12.22.00.38_veh-35_01008_01518
            │    ├── CAM_F0
            │    │     ├── c082c104b7ac5a71.jpg
            │    │     ├── af380db4b4ca5d63.jpg
            │    │     ├── ...
            │    │     └── 2270fccfb44858b3.jpg
            │    ├── CAM_B0
            │    ├── CAM_L0
            │    ├── CAM_L1
            │    ├── CAM_L2
            │    ├── CAM_R0
            │    ├── CAM_R1
            │    ├── CAM_R2
            │    └──MergedPointCloud
            │         ├── 03fafcf2c0865668.pcd
            │         ├── 5aee37ce29665f1b.pcd
            │         ├── ...
            │         └── 5fe65ef6a97f5caf.pcd
            │
            ├── 2021.06.09.17.23.18_veh-38_00773_01140
            ├── ...
            └── 2021.10.11.08.31.07_veh-50_01750_01948

Lastly, you need to add the following environment variables to your ~/.bashrc according to your installation paths:

export NUPLAN_DATA_ROOT=/path/to/nuplan/data/root
export NUPLAN_MAPS_ROOT=/path/to/nuplan/data/root/maps
export NUPLAN_SENSOR_ROOT=/path/to/nuplan/data/root/nuplan-v1.1/sensor_blobs

Or configure the config py123d/script/config/common/default_dataset_paths.yaml accordingly.

Installation

For nuPlan, you need to additionally install the nuplan-devkit and optional dependencies in py123d. You can install both either from PyPI or from source:

pip install py123d[nuplan]
pip install "nuplan-devkit @ git+https://github.com/motional/nuplan-devkit/@nuplan-devkit-v1.2"

pip install -e .[nuplan]
pip install "nuplan-devkit @ git+https://github.com/motional/nuplan-devkit/@nuplan-devkit-v1.2"

Conversion

You can convert the nuPlan dataset (or mini dataset) by running:

py123d-conversion datasets=["nuplan_dataset"]
# or
py123d-conversion datasets=["nuplan_mini_dataset"]

Dataset Issues

  • Map: The HD-Maps are only available in 2D.

  • Camera & LiDAR: There are synchronization issues between the sensors and the ego vehicle state.

  • Bounding Boxes: Due to the auto-labeling process of nuPlan, some bounding boxes may be noisy.

  • Traffic Lights: The status of the traffic lights are inferred from the vehicle movements. As such, there may be incorrect labels.

Citation

If you use nuPlan in your research, please cite:

@article{Karnchanachari2024ICRA,
  title={Towards learning-based planning: The nuplan benchmark for real-world autonomous driving},
  author={Karnchanachari, Napat and Geromichalos, Dimitris and Tan, Kok Seang and Li, Nanxiang and Eriksen, Christopher and Yaghoubi, Shakiba and Mehdipour, Noushin and Bernasconi, Gianmarco and Fong, Whye Kit and Guo, Yiluan and others},
  booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
  year={2024},
}
@article{Caesar2021CVPRW,
  title={nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles},
  author={Caesar, Holger and Kabzan, Juraj and Tan, Kok Seang and Fong, Whye Kit and Wolff, Eric and Lang, Alex and Fletcher, Luke and Beijbom, Oscar and Omari, Sammy},
  booktitle={Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year={2021}
}