nuScenes

The nuScenes dataset is multi-modal autonomous driving dataset that includes data from cameras, LiDARs, and radars, along with detailed annotations from Boston and Singapore. In total, the dataset contains 1000 driving logs, each of 20 second duration, resulting in 5.5 hours of data. All logs include ego-vehicle data, camera images, LiDAR point clouds, bounding boxes, and map data.

Overview

Papers	nuscenes: A multimodal dataset for autonomous driving
Download	nuscenes.org
Code	nuscenes-devkit
License	CC BY-NC-SA 4.0 nuScenes Terms of Use Apache License 2.0
Available splits	nuscenes_train, nuscenes_val, nuscenes_test, nuscenes-mini_train, nuscenes-mini_val, nuscenes-mini_test
Interpolated splits (10 Hz)	nuscenes-interpolated_train, nuscenes-interpolated_val, nuscenes-interpolated_test, nuscenes-interpolated-mini_train, nuscenes-interpolated-mini_val

Available Modalities

Name	Available	Description
Ego Vehicle	✓	State of the ego vehicle, including poses, dynamic state, and vehicle parameters, see EgoStateSE3.
Map	(✓)	The HD-Maps are in 2D vector format and defined per-location. For more information, see MapAPI.
Bounding Boxes	✓	The bounding boxes are available with the NuScenesBoxDetectionLabel. For more information, see BoxDetectionWrapper.
Traffic Lights	X
Pinhole Cameras	✓	nuScenes includes 6x PinholeCamera: PCAM_F0: CAM_FRONT PCAM_R0: CAM_FRONT_RIGHT PCAM_R1: CAM_BACK_RIGHT PCAM_L0: CAM_FRONT_LEFT PCAM_L1: CAM_BACK_LEFT PCAM_B0: CAM_BACK
Fisheye Cameras	X
LiDARs	✓	nuScenes has one LiDAR of type LIDAR_TOP.

Dataset Specific

class py123d.conversion.registry.NuScenesBoxDetectionLabel

Semantic labels for nuScenes bounding box detections. [1] https://github.com/nutonomy/nuscenes-devkit/blob/master/docs/instructions_nuscenes.md#labels

VEHICLE_CAR = 0

VEHICLE_TRUCK = 1

VEHICLE_BUS_BENDY = 2

VEHICLE_BUS_RIGID = 3

VEHICLE_CONSTRUCTION = 4

VEHICLE_EMERGENCY_AMBULANCE = 5

VEHICLE_EMERGENCY_POLICE = 6

VEHICLE_TRAILER = 7

VEHICLE_BICYCLE = 8

VEHICLE_MOTORCYCLE = 9

HUMAN_PEDESTRIAN_ADULT = 10

HUMAN_PEDESTRIAN_CHILD = 11

HUMAN_PEDESTRIAN_CONSTRUCTION_WORKER = 12

HUMAN_PEDESTRIAN_PERSONAL_MOBILITY = 13

HUMAN_PEDESTRIAN_POLICE_OFFICER = 14

HUMAN_PEDESTRIAN_STROLLER = 15

HUMAN_PEDESTRIAN_WHEELCHAIR = 16

MOVABLE_OBJECT_TRAFFICCONE = 17

MOVABLE_OBJECT_BARRIER = 18

MOVABLE_OBJECT_PUSHABLE_PULLABLE = 19

MOVABLE_OBJECT_DEBRIS = 20

STATIC_OBJECT_BICYCLE_RACK = 21

ANIMAL = 22

to_default()

Inherited, see superclass.

class py123d.conversion.registry.NuScenesLiDARIndex

NuScenes LiDAR Indexing Scheme.

X = 0

Y = 1

Z = 2

INTENSITY = 3

RING = 4

Download

You need to download the nuScenes dataset from the official website. From there, you need the following parts:

• CAN bus expansion pack

• Map expansion pack (v1.3)

• Full dataset (v1.0)

  • Mini dataset (v1.0-mini) (for quick testing)

  • Train/Val split (v1.0-trainval) (for the complete dataset)

  • Test split (v1.0-test) (for the complete dataset)

The 123D conversion expects the following directory structure:

$NUSCENES_DATA_ROOT
  ├── can_bus/
  │   ├── scene-0001_meta.json
  │   ├── ...
  │   └── scene-1110_zoe_veh_info.json
  ├── maps/
  │   ├── 36092f0b03a857c6a3403e25b4b7aab3.png
  │   ├── ...
  │   ├── 93406b464a165eaba6d9de76ca09f5da.png
  │   ├── basemap/
  │   │   └── ...
  │   ├── expansion/
  │   │   └── ...
  │   └── prediction/
  │       └── ...
  ├── samples/
  │   ├── CAM_BACK/
  │   │   └── ...
  │   ├── ...
  │   └── RADAR_FRONT_RIGHT/
  │       └── ...
  ├── sweeps/
  │   └── ...
  ├── v1.0-mini/
  │   ├── attribute.json
  │   ├── ...
  │   └── visibility.json
  ├── v1.0-test/
  │   ├── attribute.json
  │   ├── ...
  │   └── visibility.json
  └── v1.0-trainval/
      ├── attribute.json
      ├── ...
      └── visibility.json

Lastly, you need to add the following environment variables to your ~/.bashrc according to your installation paths:

export NUSCENES_DATA_ROOT=/path/to/nuplan/data/root

Or configure the config py123d/script/config/common/default_dataset_paths.yaml accordingly.

Installation

For nuScenes, additional installation that are included as optional dependencies in py123d are required. You can install them via:

PyPI

pip install py123d[nuscenes]

Source

pip install -e .[nuscenes]

Conversion

You can convert the nuScenes dataset (or mini dataset) by running:

py123d-conversion datasets=["nuscenes"]
# or
py123d-conversion datasets=["nuscenes-mini"]

Note

The conversion of nuScenes by default does not store sensor data in the logs, but only relative file paths. To change this behavior, you need to adapt the nuscenes-sensor.yaml or nuscenes-mini.yaml converter configuration.

Interpolated Conversion (10 Hz)

The standard nuScenes dataset provides keyframe annotations at 2 Hz (every 0.5 s). The interpolated converter upsamples this to 10 Hz by leveraging the intermediate sensor sweeps that nuScenes records between keyframes. You can convert the interpolated variant by running:

py123d-conversion datasets=["nuscenes-interpolated"]
# or
py123d-conversion datasets=["nuscenes-interpolated-mini"]

The interpolated conversion uses the NuScenesInterpolatedConverter.

Interpolation Details

Frame selection. The nuScenes LIDAR_TOP sensor records sweeps at approximately 20 Hz. The converter collects all lidar sample_data records (keyframes and non-keyframe sweeps) for a scene, then selects a subset at approximately 10 Hz by placing regular target timestamps between each pair of 2 Hz keyframes and picking the closest lidar sweep for each target. All original keyframes are always included.

Ego pose. Every lidar sweep (including non-keyframe sweeps) has its own ego_pose record in nuScenes. The converter uses these real ego poses rather than interpolating between keyframes. Dynamic state (velocity, acceleration, angular velocity) is obtained from the CAN bus by matching the closest CAN bus message to the sweep timestamp.

Bounding box interpolation. Bounding box annotations only exist at 2 Hz keyframes. For intermediate frames the converter interpolates between the surrounding keyframe annotations:

• Detections are matched across consecutive keyframes by their instance_token (track ID).

• Position (x, y, z): linear interpolation.

• Rotation (quaternion): spherical linear interpolation (SLERP) via pyquaternion.

• Dimensions (length, width, height): linear interpolation.

• Velocity: linear interpolation.

• Detections that only appear in one of the two surrounding keyframes (track starts/ends) are excluded at interpolated frames and only written at their actual keyframe.

• num_lidar_points is set to 0 for interpolated frames.

LiDAR. Each selected 10 Hz frame uses the actual lidar point cloud file from the corresponding sample_data sweep, so no point cloud interpolation is performed.

Cameras. At keyframes, cameras are extracted as in the standard converter (using the sample["data"] references). In nuScenes, these references point to the camera image captured just before the lidar sweep completes, aligning the camera observation to the end of the lidar sweep. At non-keyframe timestamps the converter follows the same convention: for each camera channel it selects the most recent sample_data record whose timestamp is at or before the lidar sweep timestamp, within a 100 ms tolerance (one full ~12 Hz camera period), consistent with the keyframe extraction.

Note

The interpolated converter requires the same nuScenes data as the standard converter, including the sweeps/ directory which contains the non-keyframe sensor data.

Dataset Issues

• Map: The HD-Maps are only available in 2D.

• …

Citation

If you use nuScenes in your research, please cite:

@article{Caesar2020CVPR,
  title={nuscenes: A multimodal dataset for autonomous driving},
  author={Caesar, Holger and Bankiti, Varun and Lang, Alex H and Vora, Sourabh and Liong, Venice Erin and Xu, Qiang and Krishnan, Anush and Pan, Yu and Baldan, Giancarlo and Beijbom, Oscar},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  year={2020}
}