Navigating the Future

Most animals, including humans, need to be able to navigate around their environment to find food and water, to return to shelter and learn where there is danger. This means that our brains are astonishingly adept at working out where we are. Kate Jeffrey is a neuroscientist at UCL and she spoke to Georgia Mills about how our brains can pull this off... 

Kate - Well a very important thing when you're walking round a familiar environment is memory. So you've been there before, for example you come out of the underground tube station and there's the street. The first thing you brain has to do is recall, based on what you can see now, its map of the last time you were there and that map has an orientation to it, so a compass direction which you've learned about as well and then you have to decide where you want to go. So somewhere in that map is a goal so, so for example, if you want to go to the pub or the post office or something. So you have to try and recall in your memory where that goal is and then plan a path from one to the other.

Georgia - And what form does this map take in the brain?

Kate - Well, like all forms of knowledge, it's fundamentally activity of neurons. And we think that some of the neurons that are particularly important for space are these things called space cells, which are found in the hippocampus which is a little part of the brain, very deep in the part of the brain behind the ears - the temporal lobe. These neurons become very active when you go to a particular place. So each neuron has it's own place where it becomes active and, we think, what these neurons are doing is signalling that you're at a particular place. And often it's a place that you've been to before, in which case it will be the same neuron that was active the last time you were there.

Georgia - Okay. So, for example, when I go into my kitchen, would my kitchen neuron light up?

Kate - That's right, although that's not just one neuron - its many. So as you're walking around your kitchen, these neurons will become active one after the other. The map is spread among hundreds of thousands of neurons - it's not just one. But the puzzle now is a neuron being active might tell you where you are, but how do you represent where you want to be? So is there another map or is the same map and, if so, what does the activity of the place where you want to be look like? So we haven't solved that one yet - it's still a mystery.  

Georgia - As well as these place cells, which light up when you're in a familiar area, are even more mysterious neurons which can help you navigate around any space, and they're called grid cells!

Kate - Grid cells are neurons that are found in the entorhinal cortex, which is the part of the brain that sends most of the cortical projections to the hippocampus from the sense organs and so on, and these neurons were investigated because of the this projection. So it was thought that if we look at what was sending to the place cells, we might get some insights as to how the place cells know where the rat is, or the human.

And it was discovered that these neurons are also active when the rat goes into specific places. But it's not just one place for these cells, it's multiple places, and these multiple places are evenly spaced and regularly ranged. The neurons themselves are randomly scattered throughout the entorhinal cortex, but the patches of activity are equally spaced. It's hard to imagine quite what that means if you haven't seen a neuron but, when the rat goes into a particular place, a grid cell will be active. If the rat walks 30 cm north, let's say, that cell will become active again. The rat walks another 30cm north and that cells becomes active again. And if you plot out all of the places this neuron likes to fire, you find it makes this really regular pattern. So it's a very extraordinary discovery.

It's a type of thing that we would never have predicted in a million years but, once it was discovered, it immediately all fell into place - aha, this how the place cells know how far the rat has walked. It's using these grid cell, and these grid cells are switching on and off, and on and off, and on and off as the rat walks and that's how the place cells know how far the rats gone. So now we're trying to understand exactly all of that works.

Georgia - Yeah, that's incredible. I'm assuming it doesn't matter how fast or slow the rat is going, they seem to be able to know the distance. Do we have any idea how these neurons are lighting up so consistently?

Kate - Not yet. There are lots of theories and you're quite right, it's independent of how fast or slowly the rats walking, so the cells are able to interpret the sensory information that's coming in based on how fast they know the rat's going. So, somewhere in the brain is some neurons that are calculating how fast the rat's going, so the hunt is on now to find out where those are. They may be in the entorhinal cortex; there's some evidence for that, but then the question becomes well, how do they know how fast the rat's walking? And there's a whole lot of ways that might be: from the visual world and the speed with which it moves past the eyes, from the movements the rat's making, how fast the legs are moving, and so on, and so on. So a lot of information is being integrated to make these calculations. 

Georgia - Our brains, it turns out, can layer grids over our environment while we move - pretty impressive. But, with all our reliance on our smartphones and satnavs to get us around, will our navigation centres, our hippocampuses, shrivel away and die.

Kate - That's a very good question. Is it really use it or lose it? So I think it's probably too soon to tell the answer to that. My hunch is probably not, because the hippocampus is very busy. You're using it all of the time just to walk around your house, even though you don't need your phone and GPS or anything. You still needing your hippocampus to know where you are in your house; how do I get from my bedroom to my bathroom and so on, so it's always busy?

Maybe it's not as busy these days as it would have been in olden times when we were navigating through the forests. That's quite possibly true but, on the other hand, we stimulate our brains in a lot of other ways with our watching television, and our virtually reality games, and so on. So I think we're not going to become a species of decrepit, brain damaged people unable to function. I think we'll adapt; we always adapt and technology always ends up enriching our lives, I think.     

These days, probably rely heavily on GPS. This stands for Global Positioning System, but how many of us actually know how it works? It was actually created by the US Department of Defence, and it's one of several branches of GNSS - or global navigation satellite systems. Terry Moore is Professor of Satellite Navigation at the University of Nottingham, and he got Georgia Mills up to speed on how having satellites around the world can tell me - and my smartphone - where I am...

Terry - You are positioning yourself with respect to the satellites. Now there's a general misconception about this, and this is sometimes popularised by the press and films and things, that it's a tracking system - the sort of "spies in the sky" are watching you all the time! When you think, currently, at the moment, there are four billion receivers in the world, so it clearly doesn't work like that!

So, what happens is, signals leave the satellites and they're transmitted down, just like a radio signal on the radio or the TV. So they're just transmitter signals and then your unit in your smartphone or your satnav in your car receives those signals. And what we actually do - it's all based on time. So the satellites have very precise clocks and they transmit coded signals that basically say when the signal left the satellite. And what your receiver does in your smartphone is it times when those signals arrive, and by knowing when it arrived, and when it left, we can work out the distance using the speed of light. So, by measuring the distance to all these different satellites, we can just do some geometry and work out where we are.

Georgia - Right. So I guess it's very important that the clocks are accurate then?

Terry - Well, this is one of the really neat tricks, yeah. Because, if we think about the speed of light, if we go back to that, and we think about something that we use every day - an A4 sheet of paper. Light will travel from one end of an A4 sheet of paper to the other in one nanosecond - one billionth of a second.

So if we are wanting to position and measure the distance, let's say, to a metre, then all this timing I've talked about needs to be syncronised to three nanosecond - three billionths of a second. So we've got to have clocks on the satellite, clocks in you phone which are all going to be synchronised - that sounds impossible. Now we can get very good synchronisation on the satellites because they're using very precise atomic clocks and they're constantly monitored so we know that they are on a stable timebase.

Georgia - And time, according to Einstein, is relative. So how do you deal with the fact that how time may be moving out in space and how it may not be quite the same as how it's moving here?

Terry - Yeah, this is a very important point and it's designed into the systems. We have to take relativity into account in two ways because these clocks, as we've said, the satellites are flying along at four kilometres a second. So that has a special relativistic effect and the time appears slower. But also the clocks are flying in a different gravitational field, a much lower gravitation field than on the surface of the Earth, and so we get a general relativistic correction and that makes the clocks appear to run faster. So combine the special relativity and the general relativity together, we would get the wrong time on the clock on the satellite.

So what happens is they are actually run at the wrong rate. We seem to have them running at the rate we expect the time to be running, but they are deliberately skewed so they are running at a different rate so that by the time we see it, we're seeing the correct rate - if that makes sense.

Georgia - Without going too much into the physics, yeah. In terms of the satellites, how many are up there and how long do they last?

Terry - They are designed to have around, let's say, seven to ten years. It varies from which constellation and which generation of spacecraft, but that sort of age of about seven to ten years is a good number to work with.

And how many do you need? Well, this depends on how far away the satellites are, okay. So the satellites we use for GPS, and for the European Galileo, and the Russian Glonass, they're all about 20,000 kilometres above the Earth in what we call medium Earth orbit. And so, how many do you need for that?

Well, you need about 24, 27, 30 satellites and the different constellation of those is their key numbers. Now GPS, at the moment, currently has 31 satellites of various generations and various ages, and it's slightly overpopulated because some of the older satellites are getting quite old and they're coming to the end of their their lives. So there are newer satellites there just in case the old ones happen to have problems.

Georgia - I suppose eight to ten years doesn't actually seem that long to me when you're actually having to send something into space. I mean who picks up the bill for this?

Terry - Yes, it's an expensive business. So it depends where you are in terms of who picks up the bill. If we're talking about GPS, then the US taxpayers have been very generously doing that for us for the last thirty odd years now. So GPS is owned and operated by the United States, and it's operated as a dual-use system for civil and military uses, but it's operated by the US Air Force Space Command, so they get the money from the federal budget within the United States and that's what pays for the system.  And, as you say, it is expensive. A satellite launch may cost in the order of 60/70 million dollars.

If you are in Europe, then you and I are paying for it. Our taxes are paying for the Galileo programme, which is paid out of the European Commission's budget or the European Union's budget. And the current bill that we are paying is 900 million euros per year is being spent out of the European budget to pay for the development and deployment of Galileo.

Georgia - So GPS is already available to us in Europe, why do we need Galileo?

Terry - The European Union is very interested in developing our own system so that then we are building our own spacecraft. We are building the ground segment, we are building the receiver segment, we are building the location services industries all based on that, and that's all jobs within Europe. So you can make an argument based on market penetration. You can make arguments on what we call sovereign control.

The way that GPS is owned and operated, as I said, by a single state, the United States of America and it is there as a military force enhancer. But it is now so critical in so many aspects of our lives in terms of the positioning and navigation, but also timing. If you remember what I said earlier on, when we're solving for the way we use GPS, we solve for time all the time and that means we have a very precise way of disseminating time around. And that's now used throughout what we call critical infrastructures, so telecommunications, banking, the internet, digital radio even are all based on timing provided by GPS and other satellite navigation systems.

But that's all under the control of another state as a defence tool. So in terms of sovereign control of critical infrastructure, we need to have our hands on the system which is keeping those critical services.

As the saying goes, knowledge is power; and if you're the navigator of a naval vessel moving at over 70 miles per hour, you need to know - in an instant - where you are and where to go next. But when the cockpits of these boats were designed, no one knew whether the information being presented to the navigator on computer displays and instruments was easy to marry up with what was actually visible through the windows. Now we do, as Odd Sveinung Hareide,  manager at the Royal Norwegian Navy  Competence Centre, explained to Chris Smith.

Odd - So the navigator of a high speed craft like the Norwegian navy corvettes, which is known as the Skjold-class is quite demanding because inside the ship there is a lot of equipment. The navigator sits in a chair and in front of him he has three screens and on these screens he's got the electronic charts, the radar, and all types of sensors which are feeding through the system, so it's a complex environment to work in.  Also, of course, the world around him is moving very fast because he's travelling at speeds up to 70 miles per hour and he needs to control that the ship is in the right position all of the time. So he has to continuously monitor the position of the ship.

Chris - Therefore one would be looking out of the window, comparing that to what the charts on the screen are saying and the radar images, and trying to put all of that information through that person - man or woman's mind very quickly. How did you try to work out what the person was doing when they were doing that?

Odd - The method of collecting this information was the eye tracking glass which was glasses that you fit to the navigator. It looks like reading glasses - there are cameras that recognises where the person looks. So if he looks outside, you get a hit on the outside and if he looks at the screen, you get a hit on the screen. We did this both in sea trials and in simulator studies. We had approximately four hours of recording.

Chris - What patterns emerged - where do the people look, where do they spend most of their time staring at?

Odd - This has been a concern in the maritime community with regards to what they call the playstation mode, which is young navigators tend to rely on the information from the computer screen instead of looking out the window at the surroundings of the ship. So our worry was that a lot of the attention was with regard to the inside of the ship and displays, but what we found was that it's actually 65 percent of the time the navigator is looking outside and continuously monitoring the surroundings of the ship. But still, 35 percent of the time is used looking at something else.

Chris - Can you get a sort of finer grained analysis out in the sense that say, if I was looking at a map on my phone and then trying to work out if I was in the right place. I might look at the phone and then look back at the street and then look back at the street to see if what I thought I'd seen was what I should be seeing. Can you see those sorts of double journeys the eyes make backwards and forward between the equipment and the outside world to see how quickly people got the information and were able to process it?

Odd - When analysing the eye tracking data, and this is one of the features you can get and we call it a look back. If you, for example, look at some of the equipment inside, then you look outside and scan the environment. Maybe you look at another equipment but you look back at the same equipment, and it could be an indication of poor visibility.

Chris - Because the people who design these user interfaces and the way data is displayed, very often aren't the navigators or the pilots of these craft themselves. They are people who work with them occasionally and they present something that they think will be fit for purpose. But it's not until you actually get someone using it and you find, in fact, it's not as good as it could be.

Odd - That's a very good observation Chris, and I totally agree. It's very important issue that the engineers that are designing the systems often haven't been to sea at all. They are very good at the designing complex systems, but they have never sat in the chair that has to take the ship forward.

Chris - And are you going to sort of close that audit loop now where you've got you data, you know what they're not looking at what's going on because they're looking at a computer screen, you know when that's happening? So can, or will you now be using that to optimise the way that the cockpit is formatted and the way that the data is displayed on these computer screens?

Odd - Yes. This process is called a human centred design process. So what we are doing now, with cadets is that we have them for retrofit. So now we have collected this data and we've seen some things that need to be adjusted to make the navigator's job easier. In the retrofit process this being done and, hopefully, this will benefit the navigator in his work.

Chris - And lastly - mobile phones and people driving have got enormous attention, especially here in the UK. And people are saying it's because they're very complicated to use, very fiddly, and it takes people's attention away. It's the same problem isn't it?

Odd - It's the exact same problem and that's why it's important that we take a look and understand better how this should work, especially making design in the human into the technology remember that they have to interact.

There are so many reasons autonomous cars could change things - it could give independence to those who are unable to drive, massively reduce congestion and pollution, and also reduce traffic. But what legal and moral challenges are coming our way, and how far off are these cars from becoming a reality? Georgia Mills and Chris Smith explored the issues, starting with the technological trials we need to solve, with Oxford University's Paul Newman.

Paul - Imagine a computer was presented with an image. Now an image is literally a list of numbers.

Chris - What are those numbers?

Paul - Those numbers would be every pixel. Every small element of a picture is perhaps described by three numbers: how much red there is, how much green there is, and how much blue there is. So by writing down lists of three numbers, that's how I can code an image.

Now when we ask a machine to see, really what's happening at the fundamental level is that the machine is presented with a very, very epically long list of numbers in sets of three and we're asked to say - is that person about to cross the road?

Chris - My brain, Paul, knows how to do that, right. It does it very well. Your brain does it exceedingly well so there is a solution albeit a neuron network currently doing it in my brain. So, therefore, it must be feasible in some way to emulate this process?

Paul - Which is why I have this job. Because this is existence proof that you know, hedgehogs can do this. There is a school of thought that you look at how neural systems do it and you reverse engineer that. There's another system that you engineer it from the ground up. I'm in a luxurious position, I can believe in both.

And something different is happening now. So one of the big questions is why this excitement in machine learning and self driving cars now? What is it that's different? And I think there's a couple of things. Computer games helped us. There are things called GPUs which are Graphical Processing Units that we use to generate the amazing graphics that we see in games. And there was hardware that was developed for those machines that has made it possible to do a kind of computing that's very useful for machine learning.

Machine learning is, simply put, I'm going to give a computer a whole load of data - pictures, and I'm going to perhaps label in those pictures hedgehogs and baboons. And then the test is can that machine learn a method that when presented with another image, says "there's the baboon".

Chris - But what has that got to do with me wanting to have a car which I don't have to drive myself - I can sit in it and it will take me where I want to go?

Paul - Because we programme now machines to learn such that when presented with data about roads or cars, we don't sit down and programme a detector of cars, we label lots of data and we programme algorithms that have structure in them to allow them to learn themselves to programme a car. So we're not explicitly saying - if a red pixel next to an orange pixel, next to some chrome, above something that looks like a tyre. We would never do that because if a car point of view changed and you looked from the side, it's a different thing. It's not robust.

Chris - Now if I was talking to you as a chemical engineer and we were talking about some new drug we wanted to invent, and I ashed you how long before that's going into a patient? The stock answer is always five to ten years. So what's the stock answer for a self-driving car?

Paul - We'll start to see stuff that looks pretty compelling in around about five years, I think. In some places much before, but we have to be careful about really what is that system and what is it doing?

Georgia - There are so many reasons autonomous cars could change things - it could give independence to those who are unable to drive, it could massively reduce congestion and pollution, and also reduce accidents. But while the technology has a long way to go, there are other challenges lying ahead. For example - who's to blame if a driverless car crashes into you? Peter Lee is a lawyer and a CEO of Wavelength.LAW.

Peter - There's a myriad of related legal issues in the area of driverless cars - around privacy and around their integration onto the public highways, but also the question mark around who's liable if a driverless car crashes. And, at the moment, that still needs some work by governments and lawyers to figure that out in my view.

Some of the bigger manufacturers and companies out there, including Volvo, came out recently to say that they would be prepared to self-insure in the event of an accident. But then there's also other questions around how parliaments might legislate in this area in the near future.

Georgia - So I suppose you've got the person who owns the car, you've got the person who built the car, and you've got the people who coded the software that the car works?

Paul - Yes, that's right. So the issue in this area is in the event of an accident and the car is fully autonomous, it's not the same as if somebody was sitting behind the wheel and had been negligent. It's more complicated that and, currently, we would have to look at the chain of actors that were involved in that incident and try and apportion blame.

Georgia - And, I suppose, the issue here is that technology is moving faster than, I guess, society can keep up with. What do you think needs to happen for the world to be ready for all these technologies?

Paul - You're right, and this has happened throughout our history. Humans will push the boundaries with technology, and develop faster and faster. The law is often slower to react but, in some way, that's quite a good check and balance against technical advancement. What it shouldn't be though is a barrier. What it needs to do is protect society and protect the other individuals. It's all about creating a balance with advancing technology and existing laws.

Georgia - Peter Lee. And another thing to consider is that when you are allowing a connected system to pilot your car, this could leave it vulnerable to cyber attacks. Ross Anderson is Professor of security engineering at Cambridge University.

Ross - The biggest problem facing autonomous vehicles in terms of security is that once you start connecting complex computing devices to the internet, you have to think about security as well as safety.

In the modern connected world, once somebody discovers a combination of inputs that will cause your car to crash, they can then broadcast this to thousands of cars and, potentially, causing thousands of car crashes. And so, once a vulnerability like that is discovered and starts to be exploited, you have to patch it quickly and most car makers simply don't have any mechanism to go out and patch cars over the air the way that your laptop or your mobile phone gets patched.

Georgia - Have there been any examples so far of this kind of nefarious attacks happening?

Ross - Attacks have been demonstrated, for example, in a Chrysler Jeep in America by security researchers last year which cause Chrysler to recall and change the software in 1.4 million vehicles. What happened with the Jeep was that Chrysler used a network, which they called Uconnect, which didn't have an effective network security, and so it was possible for somebody on the network to scan all 1.4 million vehicles that used this and determined their IP addresses of their locations. So you looked at the latitude and longitude of the target vehicle and you could then integrate it, find out what it was, and then you could go in and interfere with the software.

Now the attackers then demonstrated this by taking over a Jeep in which a journalist was riding and deliberately slowing the Jeep down to the point that lorries behind him on the freeway were honking their horns and making him kind of scared.

Georgia - How do you think we can avoid this?

Ross - We're going to have to have some rules as to the safety and security certification of vehicles. This is going to be done in Europe because just as Europe makes the world's privacy rules, because Washington doesn't care and no other country's big enough to matter. So Europe is the main player in regulating the safety of all sorts of things from cars to medical devices to electrotechnical equipment. And so, fundamentally, this is a task facing the European Union to upgrade all the various agencies that are involved in inspecting various types of safety critical equipment, and issuing safety certifications for them to make sure that they're also sufficiently secure would withstand hostile attacks.

Georgia - Will it ever be possible to keep these systems perfectly safe?

Ross - Well that's an unrealistic ambition. What you actually do is you see to it when vulnerabilities come to light you've got the means to fix them quickly, and to roll out these fixes to all the millions of devices that you have in the field. That's what happens as an absolutely routine matter to all Windows kit on the second Tuesday of every month, and Android also updates your software in your phone every month. So this is just how the IT industry has evolved a way of coping with this. So there are mechanisms that work and what's needed is sellers of cars, sellers of medical devices, and so on, to get with it and figure out how to make these mechanisms work for their own products.

Georgia - And while this connectedness is what is making cars vulnerable, is also what could make them save lives. Back to Paul Newman.

Paul - I had a car accident five years ago now and I'm still haunted by it. I made a mistake - I was lucky that everyone was okay. I haven't been able to share that experience with any of my friends. That's not true about vehicles that can share their data and share their experiences.

So we can envision a world now where the accidents are always awful and yet, because of them, there are fewer of them because the mistakes and lessons learned get shared across the entire fleet. So you can imagine a world where a fender bender in Copenhagen actively improves someone's safety in a warehouse in Cape Town that afternoon.