By Tricia Brown
University of Iowa Strategic Communications

Imagine a neurosurgeon completing a complicated, 10-hour surgery without ever looking directly at the patient’s brain or the tumor that’s being removed.

In the world of high-tech computer engineering and augmented reality, it’s entirely possible and, in fact, is an idea that was conceptualized by a University of Iowa Health Care physician and is currently being prototyped by three undergraduate students.

The senior design project for computer engineering students Alex Young, Zach

the code to make it operational—that sends high-resolution camera images to a head-mounted display, called an Oculus Rift, worn by the surgeon. The robot changes position when the surgeon moves his hand over a Leap Motion sensor—a device similar to a mouse but doesn't require touch—allowing him to see from a different angle without shifting position or touching the robot.

Augmented reality functions will project patient data, such as an MRI or 3–D image of a tumor, into the surgeon’s field of vision inside the Oculus Rift along with the images projected from the camera.

When Young, Mennen, and Tollefson—who are all from Iowa—were considering a project idea, they said they wanted to work on something that would help people. Sure, it might sound cliché, but their goal was to tackle a more advanced project, one more meaningful than a smartphone app.   

“It has real-world-type applications that are really going to change the way that people operate today,” Young says of the robot.

That’s precisely what Dr. Oliver Flouty, a third-year neurosurgery resident physician at UI Hospitals and Clinics, was thinking when he approached the director of UI’s Center for Computer Aided Design with the idea in March.

Flouty wants to replace the microscope that’s currently used to view the brain during surgery with technology that would allow him to remain comfortably seated while a robotic arm moves to provide view into a very small opening instead of the surgeon standing in an awkward position for hours in order to operate from a particular angle.

“The problem with the microscope is it makes the surgeon himself tethered to that scope because he has to look into it and at the same time operate on the patient,” Flouty says. “So, with long surgeries, say nine or 10 hours or more, it becomes stressful on the body.”

So he approached Karim Abdel-Malek, director of CCAD, with his idea for a robotic arm that would project camera video into each eye separately, giving the surgeon binocular vision. Abdel-Malek took the idea a step further: If you’re using camera video, why not enhance the video to project MRI, CT, vascular images, or any other patient data into the field of vision?

Young, Mennen, and Tollefson spent the fall semester integrating the various technologies so they function as a single unit and writing code to enable them to do so. After Young and Mennen graduate in December, the project will be handed off to a new team for additional work, Abdel-Malek says.

But the experience of getting a project of this importance off the ground has been invaluable to the students.

“When Flouty showed us what they’re using now, it’s so outdated,” Young says. “The giant microscope isn’t exactly new technology, so bringing in new technology is really cool.”

“It is really interesting,” Mennen says. “I started in biomedical engineering, so I thought maybe if I wanted to I could get into this kind of stuff. This project has meant learning a lot of new things, but it’s been fun.”

Flouty says he got the idea for the robot when he began thinking about how beneficial it would be if the surgeon didn’t have to follow the microscope around in order to see from every angle.

“What if the microscope could be independent of you so you can just sit and operate very comfortably for hours and maintain your stamina throughout the procedure,” Flouty says.

The engineers have spent dozens of hours on the project, meeting mostly on Tuesday and Friday afternoons throughout the semester at the Engineering Research Facility. Writing code and incorporating the technologies has been challenging, they say, and requires a lot of trial and error.

But they see immense value in having worked on a project that one day could change the way surgeries are performed around the world.

“It will be interesting to see what happens in the future, when we graduate and look back, just being able to know we contributed even a little bit,” Mennen says. “It’s kind of neat.”

Flouty received his undergraduate and medical degrees from the American University of Beirut, in Beirut, Lebanon. He was attracted to the UI because of its excellent reputation for research and the resources available to support that research—case in point: the freedom he’s had to tell someone about his idea and see it come to fruition.

The robot has so much promise that patents have been filed.

“Whether or not we’re going to do it, whether it happens here at Iowa, it’s going to take place no matter what,” he says. “The question is whether we’ll be fast enough to get it done or will someone else pick it up and do it. In my opinion, it’s going to happen in the near future.”