LUMPI Track

Data

Real LUMPI data is publicly available from its original release at data.uni-hannover.de/dataset/lumpi, under its own license. We do not redistribute it — obtain it directly from the source.

See also UT-LUMPI — an UrbanTwin synthetic dataset targeting this distribution, offered as orientation that sim2real transfer is feasible.

Released for participants (starting kit)

• detection_test_frames/ — 50 real LiDAR frames (point clouds only, no labels) on which you run your detector and submit predictions.
• detection_test_frame_ids.json — the canonical, ordered list of frame IDs your predictions.json must cover.
• forbidden_frames.txt — 100 frame IDs you may not train, validate, or tune on: the union of the 50 detection-test frames and the 50 realism-reference frames. Honor system, enforced by signed declaration.
• local_eval.py — the exact scoring program shipped to the Codabench worker; reproduces server scores bit-for-bit.
• make_dummy_submission.py — produces a known-bad submission that exercises the format.
• submission_format.md — the full submission spec.

Held out (organizer-only)

• 50 real frames used as the realism reference set.
• Ground-truth boxes for the 50 detection-test frames.

Categories evaluated

This track is scored on 8 object categories: Person, Car, Bicycle, Motorcycle, Bus, Truck, Van, Unknown. The detection score is the mean AP across these categories; classes with zero ground-truth instances in the held-out set are excluded from the mean. Splits are stratified random samples with fixed seed 1234, published in the starting kit so anyone can reproduce them.

Evaluation

Two scores combine into the Combined leaderboard score:

score_realism   = mean( norm(CD), norm(MMD), norm(EMD), norm(FPD) )   (lower-is-better → 1 − clip)
score_detection = norm( AP_3d @ IoU 0.5 )                             (higher-is-better → clip)
score_combined  = 0.6 · score_detection + 0.4 · score_realism

Distributional realism

1. Your 50 synthetic frames are concatenated into one set; the 50 held-out real reference frames into another.
2. Both are masked to the point-cloud range [-40, -40, -5, 40, 40, 5].
3. Both are uniformly subsampled to 10,000 points per side (seed 1234, identical in the local evaluator).
4. One CD, MMD, EMD, and FPD is computed across the full 10,000-point sets. Lower is better for all four.

3D object detection

AP is computed over the 8 track categories at IoU 0.5 (primary) and 0.7 (secondary), using the OpenPCDet 7-DoF box format [x, y, z, dx, dy, dz, heading]. IoU is oriented-BEV rectangle overlap times Z overlap. Both KITTI 11-point and 40-point interpolated AP are reported; the 40-point form is the leaderboard primary. The KITTI truncation/occlusion difficulty filter is not applied — it depends on 2D annotations that are unnatural for roadside LiDAR.

Normalization endpoints are public in config.json, so local and server scores are identical.

Submission format

Pack everything into a single submission.zip and upload it on the Codabench Submit tab:

submission.zip
├── synthetic/
│   ├── 0001.bin … 0050.bin    # exactly 50 files, float32 N×3 (or N×4), ≤ 1.5 MB each
├── predictions.json            # keys = frame IDs from detection_test_frame_ids.json
└── declaration.pdf             # signed honor declaration (declaration.txt also accepted)

predictions.json entries look like:

{
  "<frame_id>": [
    {"box": [x, y, z, dx, dy, dz, heading], "score": 0.97, "class": "Car"},
    {"box": [...],                          "score": 0.81, "class": "Person"}
  ],
  "<frame_id_2>": []
}

• box: 7 floats, OpenPCDet convention — center (m), full extents, heading around +Z (rad).
• score: confidence in [0, 1].
• class: Person, Car, Bicycle, Motorcycle, Bus, Truck, Van, or Unknown (case-insensitive). Unrecognized classes are discarded.

Required declaration statement (verbatim, case-insensitive):

I confirm that I did not train on any frame listed in forbidden_frames.txt, and that all submitted detections were produced by a model trained only on synthetic data I generated.

Scoring typically completes in under 5 minutes; a Detailed Results view shows all component metrics including per-class AP.

Reference baselines

Reported for the Car class only, from prior work — included to show sim→real transfer is achievable on this benchmark. The leaderboard scores mean AP across all 8 categories.

Phases

Enter the LUMPI Track on Codabench →