Wiring the Brain to Robotic Limbs

Chris -   Scientists are moving closer to developing ways to interface with the brain and to decode what nerve cells are saying to each other.  Professor Andrew Schwartz is working on this at the University of Pittsburgh and he's with us now.  Hello, Andrew.

Andrew -   Hello.

Chris -   What sort of signals are you trying to eavesdrop on?

Andrew -   Well, we record from a part of the brain called the motor cortex and this part of the brain is thought to have a lot to do with controlling movements.  We're particularly interested in arm and hand movements.  So the neurons we record from seem to be related to aspects of moving the hand and arm, and particularly the direction and speed in which the hand moves.

Chris -   In other words, if you were to look at the surface of the brain and specifically the bit of the brain that we know relates to movements, there's a map of a person, or the body, on the surface of the brain and different bits of that map relate to different body parts.

Andrew -   That's right in sort of a coarse way.  That's a general way of looking at it.

Chris -   So when you are recording from different clusters of nerve cells, what does it sound like?  What sorts of conversations do nerve cells have?

Andrew -   So, we can record from individual nerve cells and the message that they send is the rate at which they fire action potentials.  So action potentials are these little impulses of electricity and the time between those impulses carries the information.  And so, what we've found is that, when you move your arm in different directions, that each neuron has a direction in which it likes to fire fastest in.  We can take account of that to try to decode the way that you intend to move your hand.  And so, over the years, we've built up a rather elaborate decoding mechanism where we can look at the direction and speed of the arm in 3-dimensional space and more recently, the angle of the wrist and now, the shape of the hand and fingers.  So we can get pretty much a complete representation of what you're trying to do with your hand by tapping into these signals.

Chris -   So rather than just turning individual muscles on and off, these motor cells in the motor part of the brain are active when they want to make a part of the body move to a certain position in 3-dimensional space.  So, in other words, if you wanted to hold your hand out at 45 degrees to yourself, there would be a bunch of cells that would fire the most when you're going to do that.

Andrew -   Right and the interesting thing is it's not just the cells that like to fire in that direction but the other cells will actually stop firing when you move against their preferred direction.  So what we do is we look at the entire population because all the neurons carry some information about that movement.  So one of the critical things about all this is that we look at the large population of cells, not just at a few individual neurons.

Chris -   It's quite clever, so you're looking at things going off as well as things turning on.  How many nerve cells do you record from?  If you want to decode meaningfully what the brain wants to do, how many electrodes do you need?

Andrew -   Well, we've used as few as 30 neurons to get a good representation of X, Y, and Z movement - 3-dimensional movement in space.  But typically, what we do now is record from 100 to 200 neurons and we get a more elaborate decoding.  So now, we can record or decode other aspects of movement as well.

Chris -   So what's the next step?  At the moment you can read those signals off, but translating that neurological chatter into something you want a computer to do or a robot to do, that's a whole different ball game, isn't it?

Andrew -   Well, yes.  Actually, the decoding principle we've known about more or less for 20 or 30 years just from doing basic research.  We study monkeys and when monkeys move their own arms, we can record this activity and then correlate the activity to the monkey's own arm movement.  Now on the last 10 or 15 years, what we've done is taken that signal and instead of using or correlating that to the monkey's own movements, we use that to drive a robot arm and the monkey can actually see that robot arm and control that robot arm without moving its own arm.  So we've sort of taken and tapped this intention out and given it a behavioural meaning to the animal now.

Chris -   And I suppose, one of the things which
Kevin [Warwick] was saying at the beginning is the brain is extremely plastic.  So even if you are not recording from precisely the right cluster of cells to do exactly what you might want to do or want to achieve, with training, the individual, be it a person in the future or monkey today, could still nonetheless learn to think along the right lines.  The brain would maybe rewire itself very slightly in order to achieve the desired movement using the electrodes you've plumed in.

Andrew -   Actually, yes.  That's one of the fascinating things.  I would word it slightly differently.  I'd say that we have to have a model of how this activity works, or what it's being used for, and our model is inaccurate.  As you said, we're only recording from a few of the billions of neurons that fire during every movement.  And so, our decode is rather inaccurate.  But what the animal is able to do is to learn the algorithm that we're using to decode this and actually help us by making his neurons fire more in line with our model.  So essentially, what the animal is doing is learning our model and helping us in that regard.

Chris -   One can obviously see that if you can decode what the brain wants to do in people or individuals who have some kind of injury or an interruption of the flow of information from the brain to the motor centres in the spinal cord that would enable them to move, you could then reconstruct movements for them or enable them to control and interact with their environment better.  What about going the other way though?  What about putting information back into the brain?  Would the same techniques you've developed to listen to neurons also enable you to talk to them?

Andrew -   Well, that's an open area of research but that's exactly what we're trying to do.  We're in process along with several other laboratories.  What that would involve in our particular case is putting sensors on the robot's fingers and joints so that we can impart tactile information and joint information back to the subject.  The way we hope to do that is by stimulating with electrical pulses again in the sensory regions of the brain to try to impart this information back to the subject.