Interpreting medical ultrasound images is a difficult task, requiring technicians to look at 2D images and mentally place a 3D representation of what the tissue looks like.
To facilitate this task, researchers at MIT have developed a new approach to ultrasound imaging that allows users to visualize 3D augmented reality images of the scanned object. Virtual reality headsets provide a 3D digital representation of exactly what objects look like in real life, making them easier to identify and analyze.
This technology could help speed up the training process for ultrasound technicians and other healthcare providers who use ultrasound. It can also be deployed in hospitals for use in tasks such as using ultrasound to place a needle in the right location for a biopsy.
This could make ultrasound more intuitive and easier to understand in training. On the clinical side, it could potentially save time, improve accuracy, and provide more peace of mind for healthcare providers. They don’t have to worry about whether they missed something. ”
Kanan Dagdeviren, associate professor of media arts and sciences at the Massachusetts Institute of Technology and lead author of the study
MIT graduate students Jason Hou and Srihari Viswanath are lead authors of the paper, which is published today. natural communication engineering. Other authors on the paper include Bowen Wu ’24 and two MIT Summer Research Program students, Dartmouth College senior Cinay Dilibal and Oberlin College senior Tanisha Shende.
3D representation
Ultrasound imaging works by bouncing high-frequency sound waves off tissue within the body and reflecting them back to an ultrasound transducer. The transducer converts these sound waves into electrical signals, which are used to create 2D images of the tissue. Ultrasound technicians are trained to convert these images into 3D mental representations of tissue.
“This is a difficult skill to learn and takes a long time to master,” Ho says. “The most difficult one is the bottleneck in mental tomography, which is trained to reconstruct 2D slices in 3D mental space. This is a cognitive burden that can lead to scan inaccuracies.”
To reduce that cognitive load, the MIT team thought it would be useful to combine two technologies: 3D ultrasound imaging and augmented reality (AR).
Three-dimensional ultrasound imaging is sometimes used in fields such as fetal imaging and echocardiography used for cardiac imaging, but most 3D ultrasound imaging systems are expensive and not widely available. In this study, the MIT team used a real-time 3D system they recently developed for use in breast cancer detection.
Their new system includes an ultrasound probe slightly smaller than a deck of cards that transmits information using a chirp data acquisition system (cDAQ). The probe contains an ultrasound array arranged in the shape of an empty square and configured so that the array can take 3D images of the underlying tissue.
This system has fewer ultrasound elements than typical 3D ultrasound systems, so it requires less power and is cheaper to build.
The data collected by the ultrasound probe is compressed and streamed to a 3D computer graphics engine called Unreal Engine. This engine converts the voxel data of ultrasound images into a direct 3D representation of the object without loss of information. When wearing an AR/VR headset, users can see this 3D rendering of the internal structure overlaid on the object’s actual location, similar to X-ray vision. By tilting the head or approaching from different directions, the user can see different views of the object, making it easier to identify it.
Now easier to use
Researchers tested a new technology called AR-VIU (augmented real-time volumetric imaging with ultrasound) on a group of 18 participants. Nine of the subjects were experts in ultrasound technology (including sonographers and physicians), and nine had never used ultrasound before.
Each user performed an identification task using four different ultrasound techniques. In one condition, they viewed a 2D image on a regular screen. This is how most ultrasound exams are done today. We also displayed 3D images on a regular screen, as well as two augmented reality conditions: one 2D and one 3D (AR-VIU).
In one experiment, users were asked to identify objects (such as springs, balls, and screws) embedded in gelatin inside an opaque container that was scanned by ultrasound. In the second set, participants were asked to use a pen to mark the location of a “tissue phantom” (a gel-like material designed to mimic human tissue). This simulates the task of finding the right place to insert the needle during a biopsy.
Researchers found that the AR-VIU system significantly improved the ability to identify and locate objects for all users. This effect was particularly strong for novices, who performed nearly as well as experts when using AR-VIU. When using a traditional 2D imaging system, experts performed significantly better than novices.
“Overlaying images with anatomy and providing 3D visual context makes ultrasound much easier to understand for beginners,” says Viswanath.
In post-experiment interviews, most novices reported that they preferred the AR-VIU approach, and many said that AR-VIU made their work easier.
“3D systems are less brain-intensive, more intuitive, and easier to understand what’s happening in the area of interest,” says Dagdevilen.
Many of the experts said they preferred traditional 2D imaging because they were familiar with it and had been trained to use it. However, these experts also said that the AR-VIU system could see benefits in some situations, such as inserting needles for biopsies or visualizing heart wall movement during echocardiography.
Researchers are currently working to further improve imaging resolution and conduct additional tests to demonstrate the accuracy of AR-VIU technology.
This research was funded by the MIT Media Lab Consortium, the National Science Foundation, the MIT HEALS Graduate Student Fellowship, and the MIT-Tata Graduate Student Fellowship.
sauce:
Massachusetts Institute of Technology
Reference magazines:
Hou, J.F.; Others. (2026). Real-time 3D ultrasound in augmented reality accelerates training and closes the performance gap between beginners and experts. Communication engineering. DOI: 10.1038/s44172-026-00692-7. https://www.nature.com/articles/s44172-026-00692-7
