Mixed Reality and healthcare: a new frontier

Mixed reality (MR) is currently the darling of immersive technology. It combines all the benefits of augmented reality (AR) and virtual reality (VR) into one amazing experience. Sagentia Innovation’s Paulo Pinheiro, Head of Software, recently managed an MR project. Here, he shares his thoughts on how the technology might be used in medical settings and applications.

What is Mixed Reality?

MR merges real and virtual worlds to produce environments where physical and digital objects co-exist and interact in real time. It’s a hybrid of reality and VR: while AR simply displays a layer of digital content on top of the real world, MR offers real-world integration and interaction with virtual content.

An emerging technology with great potential

Several landmark MR headset launches are expected over the next 24 months, from tech giants including Apple and Microsoft as well as the start-up Magic Leap. Quality, enterprise devices are hard to come by right now, but as they become more widely available, they could unlock exciting new opportunities in many aspects of healthcare.

Medical applications for MR

Learning and training

MR offers ways to revolutionise medical education. Many medical institutions already integrate it with the curriculum to provide students with valuable hands-on learning experiences.

In addition, MR could become the initial training ground for the next generation of surgical robots. Using AI, surgical robots could learn autonomous steps within a surgical procedure such as suturing.

Instant diagnoses

MR headsets can superimpose images and information on a patient’s body and even analyse a patient’s condition. The day is almost here when doctors wearing MR glasses will be able to look at a patient and instantly make a diagnosis.

Surgical applications

Pre- and post-surgery assessments stand to benefit greatly from the use of MR to optimise outcomes with more personalised interventions rooted in data analytics.

Remote surgery assistance is another promising avenue. Using MR headsets, surgeons in different locations can collaborate to assist live surgery. This concept is analogous to the use of Microsoft Remote Assist for maintenance and

repair where staff are empowered to resolve issues quickly with guidance from remote collaborators.

The technology also offers ways to enhance surgery. It provides surgical staff with new ways of accessing information that are more compatible with surgical workflows and the sterile operating room environment. For example, real-time guidance can be provided in the surgeon’s field of view during surgery via integration with surgical navigation systems and the fusion of data from multiple imaging sources.

MR glasses or screens can also project real-time information, such as blood pressure and heart rate. They can help medical professionals monitor vitals and changes in a patient’s condition better than with the naked eye too. What’s more, real-time 3D views of anatomy provide surgeons with added detail and help them make better informed decisions during procedures. As this technology matures, some of the riskiest procedures may become routine with the help of MR.

Technical considerations

Designing MR apps for medical use will inevitably present challenges. Achieving seamless integration with existing or new workflows and processes will be highly complex. MR not only provides new combinations of real and virtual worlds, but also entirely new user experiences.

Certain core principles apply when designing fluid immersive experiences. Top priorities include understanding the user’s point of view, positioning virtual objects effectively, and ensuring all parties are comfortable and safe. A virtual experience may be beautiful and immersive, but it is useless without interaction.

Identifying where in the healthcare journey MR can provide the greatest benefits, then demonstrating early concepts to key stakeholders, will be paramount for successful integration. Ultimately MR must improve patients’ medical experiences and streamline healthcare processes to deliver better outcomes.

The MR design process

Traditional interaction with digital technology is inherently 2D. With MR we interact with a 3D world. Most people are comfortable with 2D tools like pen and paper or whiteboards which often limits ideas to this dimension. Drawing in 3D is difficult for most people and using 3D tools which are built for gaming or engineering to protype ideas requires a high degree of skill.

Therefore, it is common for developers to assume responsibility for solving both technical implementation and user experience as they attempt to prototype a concept into code.

The following steps are useful in designing a MR app:

1. Generate ideas with bodystorming

Bodystorming is a great technique to quickly generate and evaluate ideas using improvised artifacts. Using simple and cheap crafting materials, physical props are built to represent digital objects, user interfaces and animations for the proposed experience.

2. Acting and expert feedback

These objects can be used to prototype the experience, staging how a user would move through specific interactions. This is the ideal time to elicit ‘expert’ feedback from stakeholders or team members. For example, if an MR experience is being designed for hospitals, acting out the

experience to a medical professional can provide invaluable feedback.

3. Capture ideas with storyboards

Feedback is used to revise the experience, then captured using storyboards for introduction to new stakeholders or potential users. A storyboard is a visual representation of the experience sequence and breaks actions down into individual panels. This technique conveys both the overall flow of an experience (at low fidelities) and the aesthetic look and feel (at high fidelities). Understanding the fidelity needs of your storyboard is key to gathering the right feedback and avoiding counterproductive discussions.

4. Asset creation and prototyping

Once the MR concept is finalised, UX designers get to work on asset creation and developers prototype the app. Developers who understand the fundamental UX concepts of MR experiences and potential design pitfalls are best placed to develop these early prototypes before embarking on full app development for release.

Regulatory matters

Since today’s MR headset manufacturers are not medical device companies by trade, it will be important to map a strategy for their use in healthcare environments. From a regulatory perspective, it should be possible to assume a similar position to that of smartphone use in a medical context. The FDA’s stance on this is that its “mobile medical apps policy does not consider mobile platform manufacturers to be medical device manufacturers just because their mobile platform could be used to run a mobile medical app regulated by the FDA”.

In March 2020, FDA launched its Medical Extended Reality (MXR) Program to address open research questions related

to the design, development and evaluation of novel MXR devices and related applications. Outcomes of this will provide a scientific foundation in support of regulatory requirements and decisions on innovative MXR devices.

Furthermore, FDA proposed a Medical Extended Reality Workgroup which is now an initiative of the public–private Medical Device Innovation Consortium (MDIC). This workgroup is developing an augmented/virtual reality implementation roadmap specifying open regulatory research questions and knowledge gaps that should be addressed to facilitate the integration of MXR devices into healthcare, accelerating device availability.

More recently, FDA has authorised the marketing of EaseVRx, a prescription-use immersive VR system that uses cognitive behavioral therapy and other behavioral methods to help with pain reduction in patients aged 18 and older with diagnosed chronic lower back pain. This exemplifies the great strides that are being made in the immersive technology space.

Conclusion

MR is quickly gaining momentum in healthcare and medicine. It can already provide high-quality assistance to doctors and medical specialists in patient treatment as well as educating consumers on how to take better care of their health. The technology holds tremendous potential for the transformation and improvement of healthcare. It will be fascinating to watch the situation unfold.