loading page

ASSIST-U: A System for Segmentation and Image Style Transfer for Ureteroscopy
  • +4
  • Daiwei Lu,
  • Yifan Wu,
  • Ayberk Acar,
  • Xing Yao,
  • Jie Ying Wu,
  • Nicholas Kavoussi,
  • Ipek Oguz
Daiwei Lu
Vanderbilt School of Engineering

Corresponding Author:[email protected]

Author Profile
Yifan Wu
Vanderbilt School of Engineering
Author Profile
Ayberk Acar
Vanderbilt University
Author Profile
Xing Yao
Vanderbilt School of Engineering
Author Profile
Jie Ying Wu
Vanderbilt School of Engineering
Author Profile
Nicholas Kavoussi
Vanderbilt University Medical Center
Author Profile
Ipek Oguz
Vanderbilt School of Engineering
Author Profile


Kidney stones require surgical removal when they grow too large to be broken up externally or to pass on their own. Upper tract urothelial carcinoma are also sometimes treated endoscopically in a similar procedure. These surgeries are difficult, particularly for trainees who often miss tumors, stones or stone fragments, requiring re-operation. One cause of difficulty is the high cognitive strain surgeons experience in creating accurate mental models during the endoscopic operation. Furthermore, there are no patient-specific simulators to facilitate training or standardized visualization tools for ureteroscopy despite its high prevalence. We propose ASSIST-U, a system to automatically create realistic ureteroscopy images and videos solely using preoperative CT images to address these unmet needs. We train a 3D UNet model to automatically segment CT images and construct 3D surfaces. These surfaces are then skeletonized for rendering and camera position tracking. Finally, we train a style transfer model using Contrastive Unpaired Translation (CUT) to synthesize realistic ureteroscopy images. Cross validation on the UNet model achieved a Dice score of 0.853 $\pm$ 0.084 for the CT segmentation step. CUT style transfer produced visually plausible images; the Kernel Inception Distance to real ureteroscopy images was reduced from 0.198 (rendered) to 0.089 (synthesized). We also qualitatively demonstrate the entire pipeline from CT to synthesized ureteroscopy. The proposed ASSIST-U system shows promise for aiding surgeons in visualization of kidney ureteroscopy.
06 Nov 2023Submitted to Healthcare Technology Letters
10 Nov 2023Assigned to Editor
10 Nov 2023Submission Checks Completed
15 Nov 2023Reviewer(s) Assigned
22 Nov 2023Review(s) Completed, Editorial Evaluation Pending
22 Nov 2023Editorial Decision: Accept