Fairleigh Dickinson Professor, Students Develop Mind/Robot Interface

By Rebecca Maxon

A quadriplegic man lies in his long-term care facility, unable to get out of bed. Yet, each morning he wakes at 8 a.m., fixes his breakfast and coffee and prepares to begin his manufacturing job with a large producer of computer chips. From his bed he operates a microscopic robotic arm and assembles the tiny computer chip components in a sterile clean room in Malaysia, 6,000 miles away. It’s not the world of today, but it could be the world of tomorrow. Research scientists around the globe, including Fairleigh Dickinson faculty and graduate students, are working to develop mind-operated computer technology so that severely disabled persons can become more functional and productive individuals.

At the University’s Metropolitan Campus, Teaneck, Assistant Professor of Computer Science Eamon Doherty teaches programmers and engineers to develop Mind-operated Devices with Robotics in this unique graduate course. The class takes students through the complete problem-solving process — from determining a need through testing new technological applications. Begun as an independent study last spring, the class initially asked students — using a robotic arm, an electronic mental/facial interface and Visual C++ computer programming, an industry standard — to enable a paralyzed person to pick up objects on a table and move them from place to place. The summer-session class’s challenge was to enable the person to pick up a snack and eat it.

It may not sound like much, but to a man who has been unable to use his body from the neck down for a decade and a half, it’s a major act of independence. “The interface enabled me to pick up two objects and move them to new locations,” Bruce Davis, a quadriplegic resident of the Cheshire Home long-term care facility in Florham Park, N.J., says of his first experience using the robotic arm. “That is something I could not do for myself in the last 15 years.” Davis has had movement of only his upper arms, neck and head since a 1984 swimming accident left him paralyzed.

Rooted in Ingenuity

This fascinating research project came to FDU in fall 2001 with Doherty, who began researching ways to enable the disabled in 1997 with his doctoral thesis, “An Investigation of Bio-electric Interfaces for Computer Users with Disabilities,” completed in June 2001 at the University of Sunderland, England.

Doherty and Davis met in 1989 at the County College of Morris, where they worked in the mathematics department. Doherty offered to teach Davis amateur “Ham” radio operation, which led to their continuing friendship. Davis, who as a paraprofessional in the college’s Center for Assessment and Learning tutors mathematics three days a week, has been a willing test subject and commentator/co-author for much of Doherty’s research on mind-operated robotics.

When Doherty discussed his research with his computer programming students, he found that many shared an interest in C++ programming and in robotics. And, “There were students who wanted to develop some type of application to help handicapped people,” he recalls. So, with department approval, Doherty offered an independent study in mind-operated robotics last spring. The subject was introduced formally as a course this past summer.

Students were asked to write Visual C++ programming to enable the user to move an object on a table with a robotic arm controlled through a mind-operated interface mechanism, Cyberlink. Based on technology that senses and responds to minute surface electrical signals transmitted by brain and facial muscle activity, Cyberlink enables hands-free control of a computer, allowing users to point the cursor and “click” using a facial movement or even intense concentration. Cyberlink is commercially available and has been used to help severely disabled persons to communicate more easily using the computer. Ali Azhar, a graduate student of computer science from Morris County, N.J., who attended Mind-operated Devices with Robotics this summer, says, “I was particularly interested in how the brain signals were used to control the computer.”

Students designed an interface to enable the user, through the computer, to operate a robotic arm built by the students from a commercially-available kit of about 200 parts. “The user selects a button from a menu of buttons by scanning a computer screen with his brain waves being collected by an electrode as a signal,” explains Azhar, a senior engineer with Schindler Elevator Corp., Morristown, N.J. “A signal generator amplifies the signal and feeds this signal to a computer. The computer then sends commands to [a robotic arm through] an interface board connected to its parallel port.”

Thus, the class challenges students to take existing technology a step further. Doherty explains, “The union of these things is new, and the program that brings them together is new — and that’s what the students are writing.” Doherty’s summer students further automated the device using more advanced programming, allowing the user to pick up a snack and eat it.

In addition to computer programming, Mind-operated Devices with Robotics introduces students to a realm of possible areas in which they can use their computer expertise. Lan-Hsiang Huang, an international graduate student from Kaohsiun, Taiwan, who is studying for an MS degree in computer science, attended both the independent study and this summer’s formal class. “Before I took this class,” she says, “all the programs that I wrote were only for building a basic concept. This class gave me an opportunity to apply my concepts.”

Riveted by Research Science

Doherty’s course teaches much more than computer programming. Students learn to evaluate the test subjects’ needs; assemble a robot; build a computer circuit board; design an effective interface; and to test, evaluate and report their research results. And, since they are conducting research involving human test subjects, all students in the class become certified researchers through the National Institutes of Health, receiving online training in human-subject research as well as official recognition as a researcher, an added benefit.

“We’re doing everything in class,” says Doherty, “application, research and testing. I teach them things as varied as how to talk to a research subject with respect, the physical function of the brain, how to solder and what safety equipment to use, proper procedures for handling the equipment and personal hygiene to avoid getting anybody sick.”

After collecting information from paralyzed individuals, their families and nurses, the students “use what’s known as ‘systems analysis and design’ to make paper models of the information they collected and to determine what the paralyzed person wants to do.” Then each student or group builds the robotic arm as well as designs and constructs a circuit board for the interface.

Resolved Through Persistence

Students keep research journals following the progress of their projects and learn to write a formal research report: “This is what I tried to do, this is what I did, this is what worked, this is what didn’t work, this is what I’d do different in the future,” Doherty explains.

Upon testing their system in the laboratory, Huang and her partners found that the user’s needs encompassed more than simply picking up a cookie. “It was very difficult to get one snack from a plate,” she noted. To overcome this hurdle, the group designed a feeding system they called the Snack Slide, which delivers a cookie vertically to the end of a slide that is positioned so the robotic arm can easily grasp the snack. When the arm removes one cookie, another is delivered.

The most challenging part of the project for Huang, however, was the computer interface. The difficult aspect, she recalls, was to set the cursor to scan the on-screen buttons one by one, which would allow the user sufficient time to click on a desired action.

For Azhar, the timing of the robotic arm was a crucial hurdle. “The strength of the batteries or a drift in the gears of the robot would affect the timing,” he recalls, leaving the potential for the robot to be in the wrong position when receiving its next command.

As in most experimental research, there are monkey wrenches in the mix. Huang reports, “We found that, when Mr. Davis chewed the cookie, he also clicked other buttons because the sensor switch still transferred the signal to the system.” The program was then modified to move the cursor away from the buttons when the user chews. And so the cycle of experimentation continues — and is applied.

Huang says she learned many things in developing the interface. “This was the first time I tested a program with a human subject,” she says. “It also taught me how to talk to someone who is paralyzed and how to design a system that considers what the user needs. I had hoped I could design a system that is truly helpful in the real world,” Huang says. “This class exceeded my expectations.”

Azhar says he was highly motivated by the prospect of helping disabled people, and he learned “about the necessities and life” of the disabled. But the rewards are not just for the students. Azhar was highly impressed by Davis’ enthusiasm for the project.

Davis says, “I enjoy working with students. I also am learning as we go along. It’s most rewarding to know I’m part of fundamental research in an area that will help disabled people in the future.”

“Of course,” says Doherty, “we want to help the greater community through the development of this tool. This is an important component of technology: Technology is not developed for its own sake, but to serve people in a good way — in this instance, to help a paralyzed person regain a small degree of independence.”

Racing Toward Tomorrow

Doherty has presented the concept and prototypes to people in the medical manufacturing industry and hopes that one day a health-care company will “see the system work and say, ‘Hey, I’ll take that a step further and make a product to do this.’ That would improve a lot of people’s lives.”

A robotic arm was not left for Davis “because it was a prototype and still in the experimental stage,” according to Azhar. In the meantime, the class may move the project into cyberspace, with a robotic arm at Fairleigh and a paralyzed user at Cheshire Home using the Internet to operate it. Discussions are ongoing with an industry leader and may lead to the donation of industrial-quality robotic arm equipment to the University for further research efforts. This semester, Doherty’s students worked on a hands-free telephone-dialing system for the disabled. Future work may involve a robotic video monitoring system to enable people caring for Alzheimer’s patients in their homes to monitor them as the patients move from room to room.

Huang hopes to build an actual Snack Slide out of plastic and, even with the class done, to further develop the mind-operated system. “I think once the system is modified, it may be used for other paralyzed patients,” she says. After graduating in May 2003, she plans “to develop some drafting and graphing systems that can help people who have difficulty with movement to draw.”

As a senior engineer, Azhar hopes to be able to take on an increased level of responsibility when he gets his master’s in 2005, and “to be able to tailor my designs toward the needs of disabled people.”

But how could this one class and this one professor do so much to prepare a person for a career? “I learned how to determine the customer’s requirements and to develop a design. I was able to develop all phases of a prototype. And, I now know how to write a comprehensive research journal,” Azhar reports. All this, plus gaining a working knowledge of Visual C++.

“What I tell people,” Doherty says, “is you’re learning enough to do the job — to solve a problem — just like you would if you were working in the field. That’s the kind of emphasis we’re trying to give to the students. You could spend a year learning just a bit about the brain; but the point is we’re learning enough about the basic lobes of the brain and some of the signals they produce so we know how the mental interface works and some of the problems that the user might have with it.”

And that’s not all. Fairleigh Dickinson University students in the spring independent study put another feather in their caps by co-authoring an academic paper with Doherty, which he presented at the University of Sunderland in England to a group of professors, roboticists and people who work with the disabled. “The paper is on the Internet now for people to review and learn from,” Doherty says. (See www.headtrauma.com/sum02.htm Click Scientific Paper for England.)

Doherty will teach the class again next spring and looks forward to building on the groundwork laid by the past sessions. “What I’m hoping to tell people in this class,” Doherty says, “is that once they graduate, this isn’t over. There are companies, like Robotic Motion Inc., that have doctors, computer scientists and programmers working together.”

Having been there from the start, Doherty is proud of the way his thesis project has grown — and continues to grow. “It’s as if we’re growing along with it,” he says.

“My next goal is to attach paint brushes and try to give artists who become paralyzed a means to paint again.” Doherty has a volunteer in Englewood, N.J., with whom he would work on this application.

Perhaps this technology, once revised and perfected, could offer a whole new realm of possibilities — not only for occupational opportunities, but for a range of self-care. Imagine a sophisticated mechanism that can help immobilized users to brush their teeth, comb their hair, even dress themselves. The concept has been demonstrated here. It can happen. And, with the dedication of scientists such as Doherty, Huang and Azhar — and many more to follow in their paths — it will.

FDU Magazine Directory | Table of Contents | FDU Home Page | Alumni Home Page | Comments