2
Robotic platforms (Raven-II, da Vinci Xi)
32
Trials (15 peg transfer, 17 suturing)
Multimodal
EM hand tracking, RGB-D hands, surgical video, foot pedals, robot states
Open & Non-Invasive
Open-source, platform-agnostic system with no proprietary hardware access required
Purpose: Robot-assisted minimally invasive surgery (RMIS) research increasingly demands multimodal data, yet access to proprietary surgical robot telemetry remains a critical barrier. This work introduces MiDAS, an open-source, platform-agnostic system designed to enable time-synchronized, non-invasive multimodal data acquisition across diverse surgical platforms.
Methods: MiDAS integrates external sensors, including electromagnetic (EM) hand tracking, surgical video, RGB-Depth (RGB-D) hand tracking, and foot pedal interactions, without requiring proprietary hardware access. We validated the system on both the open-source Raven-II and the clinical da Vinci Xi, collecting datasets during peg transfer and hernia repair suturing tasks. We conducted correlation analysis to quantify how well external EM tracking approximates internal robot kinematics and performed downstream gesture recognition experiments with modality ablation studies.
Results: Correlation analysis confirms that EM hand tracking closely approximates robot kinematics for positional and rotational trajectories. Downstream gesture recognition experiments demonstrate that non-invasive motion signals (EM tracking) achieve performance comparable to proprietary robot kinematics. Furthermore, visual streams are shown to benefit significantly from domain-adaptive and self-supervised pretraining strategies.
Conclusion: MiDAS enables accurate, extensible, and reproducible multimodal data collection for surgical robotics research across both open and commercial platforms. The system successfully lowers barriers to data-driven learning in RMIS by providing a non-invasive alternative to proprietary data access.
Dataset & code available at https://uva-dsa.github.io/MiDAS/.
Dive deeper into the experimental setup and code on the MiDAS project page
Correlation Analysis
Quantify how well EM hand tracking approximates internal robot kinematics.
Gesture Recognition
Benchmark Transformer- and CNN-based models for gesture recognition from multimodal signals.
Modality Ablation
Identify effective combinations of motion, video, and interaction modalities for RMIS tasks.
The MiDAS data collection system and multimodal datasets are publicly released to support research in surgical robotics, gesture recognition, and multimodal learning.
The data collection rig deploys entirely external instrumentation on top of existing surgical robot consoles. A synchronization backbone aligns motion, video, and interaction streams so downstream methods can trust multimodal labels out of the box.
Open & Platform-Agnostic
Modular controllers interface with Raven-II, da Vinci Xi, or future teleoperation platforms without vendor APIs.
Non-Invasive Integration
External sensors mount to the surgeon console and OR equipment while preserving all clinical workflows.
Time-Synchronized Logging
Unified clocking aligns EM motion, RGB-D hands, stereo video, foot pedals, and robot state estimates.
@article{https://doi.org/10.1002/rcs.70220,
author = {Weerasinghe, Keshara and Roodabeh, Seyed Hamid Reza and Hawkins, Andrew and Zhang, Zhaomeng and Schrader, Zachary and Alemzadeh, Homa},
title = {MiDAS: A Multimodal Data Acquisition System and Dataset for Robot-Assisted Minimally Invasive Surgery},
journal = {The International Journal of Medical Robotics and Computer Assisted Surgery},
volume = {22},
number = {4},
pages = {e70220},
keywords = {multimodal surgical data acquisition, non-invasive motion sensing, Robot-Assisted Minimally Invasive Surgery, surgical activity datasets, surgical gesture recognition, surgical robotics},
doi = {https://doi.org/10.1002/rcs.70220},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/rcs.70220},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1002/rcs.70220},
abstract = {ABSTRACT Background Robot-Assisted Minimally Invasive Surgery (RAMIS) research increasingly relies on multimodal data, yet access to proprietary robot telemetry remains a barrier. We introduce the Multimodal Data Acquisition System (MiDAS), an open-source, platform-agnostic system enabling time-synchronized, non-invasive multimodal data acquisition across surgical robotic platforms. Methods MiDAS integrates electromagnetic and depth camera-based hand tracking, foot pedal sensing and surgical video capture without requiring proprietary robot interfaces. We validated MiDAS on the open-source Raven-II and the clinical da Vinci Xi by collecting multimodal datasets of Peg Transfer and Hernia Repair Suturing tasks performed by surgical residents. Correlation analysis and gesture recognition experiments were conducted. Results External hand and foot sensing closely approximated internal robot kinematics and non-invasive motion signals achieved gesture recognition performance comparable to proprietary telemetry. Conclusions MiDAS enables reproducible multimodal RAMIS data collection and is released with annotated datasets, including the first multimodal dataset of hernia repair suturing on simulation models.},
year = {2026}
}