HoloWrist

An FWO Research Project

Our Research Plan

Our aim is to use augmented reality techniques to display a hologram of
the patient's wrist bones on the patient themselves. The hologram
needs to be accurately aligned and move with the patient.

Flow chart of the HoloWrist Project

The goal of HoloWrist is to use augmented reality to accurately show the location of a patient's wrist bones on their wrist during surgery. This will be done by creating a hologram of the patient's wrist bones (e.g. from their CT scan) and using augmented reality headsets (e.g. Microsoft's HoloLens) to display the hologram on the patient's wrist (WP4).

To accomplish this goal requires the development of new technologies in order to accurately identify the position and pose of the wrist, then to identify the position and pose of the broken wrist bone with respect to the wrist as a whole. In HoloWrist:

New multi-camera imaging systems will be developed (WP1) to capture the location and pose of the wrist as a whole, with sub-millimetre accuracy
New ultrasound imaging techniques (WP2) will be developed that can create an ultrasound image from a flexible sensor array. This imaging will allow us to identify where the broken wrist bone is with respect to the skin surface
New motion tracking algorithms will be developed (WP3) to keep the hologram of the wrist bones aligned to the patient, with sub-millimetre accuracy, as their wrist moves during the surgery.

These items are discussed further below.

Motion Tracking
Flexible Ultrasound
Augmented Reality

Motion Tracking

Throughout a wrist surgery, the motion of the patient's wrist - and the surgical tools interacting with the wrist - need to be accurately tracked in a way that does not distract the surgeon from their task. Existing surgical tracking systems use electromagnetic probes which are cumbersome to operate and may have difficulty track small bones like the ones in the wrist.

In HoloWrist, digital 3D models of the surgical tools and patient's wrist will be created. These models will then be match to the 3D scene recorded by the new multi-camera imaging system. These cameras will be placed above the surgeon so as to not distract their work.

Flexible Ultrasound

The position of the bones relative to the skin surface can vary with the motion of the hand. An imaging technique is required to view these variations while not limiting the natural motion of the hand during the surgery. X-ray imaging (fluoroscopy) is currently used but the radiation it generates means that it can only be used a few times during the surgery.

In HoloWrist, we aim to image the wrist bones using a flexible ultrasound device, one that can move with the wrist. To do so, we will estimate the shape of the ultrasound sensor array that gives the most realistic image based on the recorded ultrasound signals. This image will then be used to identify the bone's position relative to the skin.

Augmented Reality

The integration of our motion tracking technology into an augmented reality environment requires the precise calibration of the multi-camera system, the flexible-array ultrasound device, and the augmented reality headset (e.g. Microsoft HoloLens). To date, this calibration has not been done with sub-millimetre accuracy.

In HoloWrist, we aim to develop new statistical methods and machine learning techniques to model this calibration problem with greater detail and precision. Conceptually, the cameras, ultrasound and headset will each be used in turn to help constrain the locations of the other devises. We believe that by combining the information from each device, we will achieve that sub-millimetre accuracy.

Follow Our Progress

You can follow our progress through the social media links below. This research project is a collaboration between Ghent University, KU Leuven, and the MoRe Institute. Funding is provided by the Flanders Research Foundation (FWO).