Extended Reality Spurs Healthcare Transformation

How medicine is leaning on XR technology to deliver better care

Extended reality (XR) technology has experienced a major boost during the pandemic as a result of people being unable to meet in person. Limits on our ability to travel have prompted various industry segments and roles to consider the possibilities that these technologies bring, whether they’re fully immersive virtual reality (VR) solutions or augmented and mixed reality tools that overlay digital images or information over our view of the physical world.

This is leading many to realize that these possibilities go way beyond our current situation, offering new approaches that will transform the way people live and work.

Showcasing Live Holographic Surgery

A couple of weeks ago, Microsoft hosted its 24-hour Holographic Surgery event, designed to showcase the opportunities of using XR in healthcare. With 13 live surgeries from 13 countries and a host of panel discussions, the event was the brainchild of Dr Thomas Gregory, associate professor at the Avicenne teaching hospital in France and an early pioneer of using VR in healthcare. Dr Gregory and his team now use Microsoft’s HoloLens 2 mixed reality devices in all their operating theatres, and he has been a vocal champion for XR as a transformative solution for his profession.

The event demonstrated XR technology in action, highlighting its potential in today’s environment as well as its possibilities for the healthcare sector in the future. VR has been used to support medical teaching for several years, using interactive simulations to allow trainee doctors and nurses to get hands-on experience in a safe setting, with the added benefits of better engagement, improved knowledge retention and better results in terms of their clinical performance.

In the pandemic, where people are unable to get face-to-face training, VR has provided a vital support, not just for students and junior medics, but also for rapidly allowing those in the profession to expand their skills to cope with the changing situation. My colleague Leo Gebbie highlighted the opportunities in this area in A Cutting-Edge VR Experience.

Data Visualization and In-Surgery Access to Patient Information

However, Microsoft’s event focussed not on VR, but on mixed reality with HoloLens 2 and the role the technology is starting to play in levelling up the experience of procedural operations for clinicians and patients. Dr Gregory described the technology as the control centre for the surgeon, giving them hands-free access to all the patient information they need to perform the procedure, including their records and X-ray or MRI scan images, for example.

The virtual display can be anchored to a fixed place in the surgeon’s physical surroundings, so that it’s on hand as needed but doesn’t block their view of the patient. Surgeons can interact with the information using hand gestures and voice commands, maintaining the sterile environment required in an operating theatre.

Some surgeons are already using XR to improve their interactions with patients before an operation. In this case, both the doctor and the patient wear a HoloLens headset to view the patient’s scan image in 3D, for example, so that the doctor can more clearly explain the procedure using the visual representation. This answers growing calls from patients to be more involved in their treatment plans and helps to improve their understanding of the operation, which in turn provides reassurance in the clinician’s approach. As Dr Gregory put it, “patients love surgeons that love technology”, not least because it gives patients confidence that they have access to the latest knowledge and expertise.

Improving Patient Treatment through Remote Collaboration

Recent integration of video meeting solutions like Microsoft Teams into XR devices means that surgeons can include participants from outside the theatre in the operating process. Demonstrations during the event showed specialists called in to provide opinions and collaborate on the best approach for the procedure, but the technology could also allow students to remotely observe in real-world situations. For example, the cameras in an XR headset allow trainees to see what the surgeon sees, with the benefit of being able to discuss the case with the surgeon in real time.

The use of XR for remote collaboration has grown during the pandemic, including at the Imperial College Healthcare NHS Trust in the UK, which has used the HoloLens 2 headset to help lower the number of caregivers entering Covid-19 wards. A single doctor wearing an XR device enters the ward, while other members of the healthcare team watch the video stream securely from another area, providing advice and input where needed.

The benefits from this scenario clearly extend beyond the pandemic, enabling much greater collaboration between clinicians and specialists around the globe, and potentially helping to democratize expertise throughout the healthcare sector. As one doctor explained during a panel discussion, not every hospital has a specialist for everything, so the ability to call on experts to help determine the best treatment or to walk a clinician through an unfamiliar process in real time could be transformative.

Ambitions for the Future of Healthcare

The surgery demonstrations focussed on the applications of the technology that are available today, but the panels discussions also explored longer-term opportunities for the sector. A common area of interest among healthcare experts was the overlaying of scan images on patients to help precisely guide operating strategies, for example in complex organ surgery, where it’s critical to pinpoint the exact place of the first incision. However, such solutions require specialized applications; they won’t be available “out of the box” like the Microsoft Teams-based Remote Assist application in use today.

Another aspiration of panel participants sees artificial intelligence analysing videos of procedures captured with an XR device, identifying best practices that could enable XR apps to guide surgeons through procedures.

Although we’re a long way from these types of solution being readily available, there’s clearly a growing appetite for taking advantage of XR in the healthcare sector. It’s also interesting that XR is seen as a relatively low-priced solution for existing uses; although each device costs several thousand dollars, this is considered affordable compared with specialist healthcare technology such as sophisticated robotics or even display screens. XR devices also provide much more flexibility in how and where they’re used, helping to maximize their value.

The key to this vision is building an ecosystem of software developers focussed on creating solutions to meet these needs. Of course, given the regulated nature of the healthcare industry, compliance and security factors are essential to the success of such solutions. Nonetheless, this is undoubtedly an important area of growth for the XR market, and we expect to see a swathe of new applications and uses emerging over the coming months.