This project introduces MiDAS, an open-source multimodal data acquisition system designed to support research in robot-assisted minimally invasive surgery (RMIS). The system enables synchronized collection of surgical data without requiring access to proprietary robot telemetry, making it possible to study surgical workflows and develop AI-driven assistance systems across multiple robotic platforms.

1. Platform-agnostic multimodal sensing system with a non-invasive data collection framework that captures synchronized surgical data including electromagnetic hand tracking, RGB-D hand motion capture, stereo surgical video, and foot pedal interaction signals without modifying robotic hardware.
2. Cross-platform data collection on surgical robots validated on the open-source Raven-II surgical robot and the clinical da Vinci Xi system, showing that external sensing modalities can approximate internal robot kinematics and capture surgeon interaction dynamics.
3. Multimodal surgical activity datasets with annotated data from robotic surgery training tasks, including peg transfer experiments and high-fidelity suturing procedures on realistic tissue simulation models during a surgical training bootcamp.
4. Benchmarks for surgical gesture recognition evaluating state-of-the-art temporal models for gesture recognition using multimodal signals, demonstrating performance comparable to proprietary robot telemetry.

By providing an open-source data acquisition system and multimodal datasets across multiple surgical platforms, MiDAS aims to reduce barriers to surgical AI research and accelerate the development of intelligent systems for surgical workflow understanding, skill assessment, and cognitive assistance in robot-assisted surgery.

This project introduces EgoEMS, the first high-fidelity multimodal egocentric dataset designed to support the development of AI systems that assist Emergency Medical Services (EMS) responders during critical incidents. The dataset captures realistic emergency scenarios with synchronized multimodal sensing to enable research on real-time cognitive assistance, activity recognition, and performance feedback for first responders.

1. Multimodal egocentric data collection via a synchronized sensing platform capturing responder egocentric video, conversational audio, smartwatch IMU motion signals, and CPR quality measurements using low-cost off-the-shelf devices.
2. Realistic EMS scenario simulations with over 20 hours of data across 233 simulated emergency scenarios involving 62 participants performing interventions such as CPR, defibrillation, ventilation, stroke assessment, and ECG monitoring.
3. Fine-grained procedural annotations including an EMS activity taxonomy with 67 keysteps across 9 medical interventions, timestamped audio transcripts, speaker diarization, object detection annotations, and segmentation masks for medical tools used during procedures.
4. Benchmarks for AI cognitive assistants spanning keystep classification, keystep segmentation, and CPR quality estimation using multimodal data.

By providing realistic multimodal data and standardized benchmarks, EgoEMS aims to accelerate research on AI-powered cognitive assistants that improve situational awareness, protocol adherence, and responder training in emergency medical environments, ultimately contributing to better patient outcomes.

This project aims to enhance the safety and efficiency of robot-assisted surgical procedures by combining systematic modeling of safety incidents, resilience assessment under faults and cyber threats, continuous context-aware monitoring, and simulation-based training for surgical teams:

1. Modeling and analysis of safety incidents by considering the interactions among cyber and physical system components and human operators;
2. Resilience assessment of the robotic surgical systems in the presence of accidental system faults, cyber attacks, and human errors;
3. Continuous context-aware monitoring for early detection of potential safety and security violations;
4. Simulation-based safety training to prepare surgical teams on dealing with unexpected events during surgery.

This project focuses on developing the next generation of first responder technologies that enhance situational awareness and safety in emergency response. The central goal is to design a wearable cognitive assistant system that integrates the following key components:

1. Resilient data analytics for collecting heterogeneous data streams from the incident scene, aggregating them with knowledge bases and publicly available sources, and transforming the results into accurate, actionable feedback for first responders;
2. Anytime real-time sensing and edge computing resources that are dynamically optimized to enable continuous data processing on responder-worn devices, even under unexpected events such as hardware failures or network disconnections.