EndoX: A GPU-Accelerated Endoscopic Perception Simulation Framework

Kwan Tung Lam*1,2, Ping Shu Ho*1, Patt Phurtivilai1,3, Ka Chun Cheung1
1NVIDIA AI Technology Center, NVIDIA Corporation, USA
2The Chinese University of Hong Kong, Hong Kong, China
3The University of Hong Kong, Hong Kong, China
MICCAI 2026
*Equal Contribution
Paper Code Dataset Email

Abstract

Endoscopic examination of gastrointestinal organs increasingly relies on 3D computer vision techniques such as depth estimation, surface reconstruction, and missing region detection. Yet rigorous method development for endoscopy remains challenging, as ground-truth data cannot be readily acquired during clinical procedures. We present EndoX, a GPU-accelerated synthetic data generation (SDG) framework and interactive simulator for medical endoscopy that enables efficient generation of fully annotated datasets, addressing the data scarcity gap in algorithm development. EndoX automates anatomically accurate 3D mesh generation from computed tomography (CT) data and employs physics-based rendering to produce RGB imagery together with geometric modalities: depth, surface normals, optical flow, occlusion, and camera coverage maps, with precise ground-truth geometry. For validation, we constructed organ-specific datasets by navigating a virtual capsule endoscope through anatomically accurate CT-derived organ meshes, with sensor and lighting parameters configured to approximate real capsule endoscopes. Quantitative evaluations on depth estimation, pose estimation, and 3D Gaussian splatting demonstrate that models trained on EndoX data achieve competitive accuracy, and qualitative assessment confirms transferability to clinical endoscopy. The framework is designed for scalable deployment on GPU nodes via containerization. A sample dataset comprising 3 video sequences (12,000 frames) with paired ground-truth annotations and 3D models is available. Source code of the extension for NVIDIA Omniverse, documentation, and the sample dataset are available at https://beatricelam111.github.io/EndoX/.

Contributions

We propose EndoX, a GPU-accelerated synthetic data generation framework and high-fidelity simulator for medical endoscopy that:

  • Automates anatomically accurate 3D mesh generation from CT data across diverse patient anatomies.
  • Employs physically-based rendering (PBR) for optically accurate multimodal imagery with configurable sensor, lighting, material, and trajectory parameters.
  • Is designed for scalable deployment for high-throughput data production with the containerized headless version to be publicly released upon acceptance.

Automated Generation of 3D Models

CT-to-mesh pipeline
CT-to-mesh reconstruction pipeline converting 2D CT slices into a 3D mesh model. They are imported into NVIDIA Omniverse for scene composition and serves as 3D ground truth.

Configurable Post-processing Effects

Post-processing configuration
Post-processing effects are configured to simulate optical imperfections common in clinical endoscopes and can be combined for domain randomization across synthetic frames.

Physics-Based Rendering of Arbitrary Output Variables

Normal mucosal RGB rendering Normal mucosal selected modality
DepthRGB

Normal Mucosal Texture

Pathological texture RGB rendering Pathological texture selected modality
DepthRGB

Pathological Texture

Polyp RGB rendering Polyp selected modality
DepthRGB

Polyp

A virtual capsule endoscope traverses the mesh, rendering RGB frames and dense ground-truth annotations including depth, surface normals, optical flow, and occlusion.
Coverage map animation
Camera coverage map acquisition during virtual capsule camera traversal.

Dataset and Distribution

Download Dataset on Hugging Face

The example scene in the sample dataset was reconstructed from CT colonography data (CC BY 3.0), with the mesh surface texture adopted from VRCaps. For each video sequence, we provide: (i) a triangulated 3D mesh reconstructed from CT data (Wavefront OBJ), and (ii) a vertex-wise coverage map in vertex indices array in .npz format.

For every frame, we provide the following annotations:

Depth: distance from the camera plane to the object surface, clamped to 0.1–100 mm, linearly scaled, encoded as a 16-bit grayscale image.

Surface normals: reported in world and camera coordinates; X, Y, Z components stored in R, G, B channels, each linearly mapped from [−1, 1] to [0, 65535] as a 16-bit color image.

Optical flow: per-pixel displacement between consecutive frames; R and G channels encode X- and Y-direction motion (±512 px), each linearly mapped to [0, 65535] as a 16-bit color image (the first frame carries no flow value).

Occlusion: an 8-bit binary mask within the 1–100 mm clipping range; pixels occluding other mesh faces set to 255, all others to 0.

Camera pose: per-frame world-space translation and rotation stored as tx, ty, tz, qx, qy, qz, qw (unit quaternion) in a text file.

Coverage map: vertex-wise visibility stored as an index array in .npz format.

Structure

EndoX_dataset/
├── colon/
│   ├── calibration.yaml
│   ├── sequence_1/
│   │   ├── coverage_map/
│   │   ├── rgb/
│   │   │   ├── 0000.png
│   │   │   ├── 0001.png
│   │   │   └── ...
│   │   ├── depth/
│   │   ├── normal_world/
│   │   ├── normal_camera/
│   │   ├── optical_flow/
│   │   ├── occlusion/
│   │   └── camera_pose/
│   └── sequence_2/
│       └── ...
└── small_intestine/
    ├── calibration.yaml
    ├── sequence_1/
    │   └── ...
    └── sequence_2/
        └── ...

BibTeX

@inproceedings{endox2026,
  title     = {EndoX: A GPU-Accelerated Endoscopic Perception Simulation Framework},
  author    = {Kwan Tung Lam and Ping Shu Ho and Patt Phurtivilai and Ka Chun Cheung},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)},
  year      = {2026}
}