HoloCare and Illumie Showcased at SXSW 2022

The Norwegian medical startup projects HoloCare and Illumie were present at South by Southwest in Austin, Texas.

Robert Stigliano

After a cancellation in 2020, and an online-only version in 2021, South by Southwest was back this year to it’s full, pre-pandemic glory, showcasing the trends and technologies influencing our collective future. One of this year’s themes was imagining society in the year 2050, with many presentations focusing on the potential of NFTs, blockchain, and the Metaverse. I had the privilege of sitting down with the developers of two exciting projects. They are harnessing the power of augmented reality (AR) to advance the fields of health and medicine.

At Wanderlust Wine Co. in downtown Austin, over the course of three days, Nordic creators highlighted their work through presentations and interactive demonstrations. On Wednesday afternoon, Illumie founders Ingri Skogsrød and Jelmer Verhoog stood at the podium and opened their presentation with a detailed description of their visual appearance, “I am a white female, early-thirties, wearing a black t-shirt and black jeans”, Jelmer followed suit. A curious but captivating intro, this was the lead into their presentation on Illumie, an award-winning project that uses AR technology to improve the lives of blind and visually impaired people.

Illumie was born from a global innovation competition at Sopra Steria and is intended to facilitate meaningful exploration for the visually impaired. Their primary product, ‘Shield’ runs on a smartphone, detects and identifies objects in the vicinity using LiDAR, and then relays this information to the wearer through both auditory and haptic feedback. The auditory cues are built into the existing iOS accessibility functionality that users are familiar with and thus are able to listen to, at incredible speed, providing significantly more information than by using a traditional mobility cane alone, and empowering the visually impaired to navigate their surroundings with confidence.

The second AR project presented was HoloCare, a technology that is transforming the process how surgeons see and engage with medical images. Wearing an AR headset called a HoloLens 2, Product Manager and Service Designer, Dag Otto Christensen, demonstrated how HoloCare can fuse together a 3D anatomical model of a patient’s heart with the 2D CT scan images of a patient’s body. With some hand gestures, Christensen demonstrated how the system also allows the user to turn, enlarge, and inspect the 3D model floating in front of them, a capability usually seen only in some sci-fi movies.

Through their partnership with Oslo University Hospital, HoloCare’s first application is intended to help teams of surgeons in the pre-surgical planning process. Often today in these pre-surgical treatment planning meetings, a cross-functional team of clinicians will review traditional 2D images such as CT and MRI, and 3D models that are displayed on a 2D screen. The challenge that HoloCare is looking to address is that, although surgeons are very skilled at effectively generating the 3D model in their minds, there is room for different experts to have slightly different interpretations. “By using a holographic image of the CT images to get a common view of the whole organ, clinicians can discuss the different structures, with interaction, and agree on the challenge, and how to approach it. That is really valuable.” states Christensen.

Consistent between both teams, they emphasized the importance of the direct feedback from the end user during the development process. For Illumie, this meant transitioning away from a stand-alone AR device, to instead incorporating the system into the user’s smart phone. “Having something exist and having it be nice to use are two very different things”, stated Skogsrød regarding the feedback that having another device that a user would need to carry and charge was not entirely practical. For HoloCare, Christensen identified the side-by-side collaboration with clinicians to be the most important element to ensure that what they design creates value within the clinical workflow.

Augmented reality, a technology most familiar to us now through games such as Niantic’s Pokémon GO, is fueling innovations in the health and medicine sectors. Never before a visually impaired person has been able to independently intuit their surroundings with such ease and accuracy, nor have teams of surgeons been able to so directly interact with holographic representations of their patients’ physiology, but with the power of augmented-reality and artificial intelligence these advancements are on the horizon. As stated by Prof. Wolfram Lamadé, a clinician collaborating with HoloCare, “it feels like Star Trek”, and I am excited for that future.

HoloCare is currently seeking FDA approval, and Illumie is accepting people into their pilot program on their website, https://www.illumie.no.