How did AI explode in popularity?

Artificial intelligence is being talked about and used more widely than ever. Rapid advancements in this technology have, for example, led to many remarkable and welcome breakthroughs in science, and major efficiency gains in some workplaces. It seems certain that the widespread adoption of AI is an inevitability, much like the internet and social media. But, over recent months, AI concerns have come to the forefront, both for businesses and individuals, and that’s what we’re going to explore this week. One of the main ways that the majority of us are running into AI is via platforms like ChatGPT; these are so-called “large language models”, which specialise in understanding instructions we provide in plain English and then rapidly generating summaries, explanations or even diagrams and computer code to order. And it is very impressive. But all that glitters is not gold. Because these systems make a lot of mistakes; and as a lot of Internet content is now being generated this way, we’re running the risk of polluting the knowledge space with rubbish that could potentially be around as a misleading distraction for decades. Here’s Mike Wooldridge, Professor of the Foundations of Artificial Intelligence at Oxford University…

Mike - Large language models are quite a recent development. They've really only been around in the way that we recognise them now for about half a dozen years. And what they do fundamentally is something which everybody who's listened to this programme will be familiar with. If you open up your smartphone and you start sending a text message to your mother or your partner and you say, “I'm going to be…” it will suggest completions. And how your phone is doing that is it's been trained on all of the text messages that you've sent. And it's learnt that the likeliest next thing to come after “I'm going to be,” is either “late” or “walking the dog,” or in my case, “in the pub.” Large language models have been trained, not just on a bunch of text messages, but essentially all of the digital information that's available in the world, essentially all of the worldwide web, which is an unimaginably vast quantity of text. And it is really just ordinary written human text. So every web page is scraped and all the text that's on that page is extracted from it. And all of that is fed to the enormous neural networks that are in large language models, which are trained so that they can make the best prediction possible about what should come after what we call your prompt. And your prompt might be, ‘a one-paragraph summary of the life and achievements of Winston Churchill,’ or ‘the recent performance of Liverpool Football Club in the European Champions League,’ or whatever. But all of that text, the whole of the worldwide web is absorbed by these vast, vast neural networks in a process that takes typically months, and requires millions of pounds worth of computer processing power. And what we end up with is something that's really quite remarkable.

Chris - And when these things are roving across the internet, hoovering up that information, is that indiscriminate? Or would it do it like you would do it? Which is you'd say, oh, here's a web page from the University of Oxford. I set that as having greater store and veracity and validity compared to a web page that was produced on someone's blog five years ago.

Mike - There is a certain amount of discrimination that goes on, but actually the demand for the data is so great. You really need absolutely all the data that you can throw at it, that it ends up being somewhat indiscriminate just because the demand for that data is so vast. The scale of the data is really, really astronomical. The breakthrough large language model was called GPT-3, and it was released in 2020. There were 40 terabytes of ordinary written text used to train that model. That's the equivalent of millions upon millions of ordinary books. And if your demand for that text is so great, then there's a limit to how discriminating you can be. But interestingly, the evidence is the higher quality text that you're able to train your model on, the better it's going to be. So there is a market right now for that text. There are also, I have to say, some pretty unscrupulous practices that are being used by some of the large language model builders who are, if not actually breaking the law, then probably running roughshod over copyright restrictions. And there are lawsuits underway right now about that.

Chris - So if these things are just ingesting things willy-nilly and looking for connections, if you've got these engines producing this material, which then ends up on the internet, are they going to end up force-feeding themselves their own stuff? Is that like me eating my own excrement?

Mike - Yes. Thank you for that, Chris. Let me put it this way. It seems inevitable to me that within 20 years at the most, but probably more likely within 10, essentially everything that we read on the internet, on the worldwide web is going to be AI-generated, unless you're prepared to pay a premium for human-generated content. And yes, you're absolutely right, what that means is that this is the content that models are then going to be trained on. And the evidence at the moment is that models which are trained on AI-generated content are poorer. And if you just iterate that process over a few generations, it can quite quickly descend into nonsense. So this is one of the big conundrums of the age in AI. Where are we going to get that future data from? And how are we going to ensure that models aren't just trained on ‘AI slop,’ as it's now called. But also what this signals is there is going to be a market for human-generated content. Data which is written text or whatever kind of content that's created by human beings will remain extremely valuable.

Chris - The other thing that we've heard about is that these models do tend to make stuff up. So if they are just stringing together material that they have gathered from across the internet, why do they make stuff up?

Mike - These models have no conception of what's true or false. They're simply designed to make the best guess possible about what should come after your prompt. And in particular, in the absence of any information, they will fill in gaps. But the weirdly troubling thing about this is they will fill in those gaps in extremely plausible ways. And that means that not only do they get things wrong, they get things wrong in ways that are actually quite difficult for us to detect. So here's my favourite example. An early large language model that I tried out, I asked it about me. And when we finally got to the Michael Wooldridge that wasn't the Australian Health Minister or the BBC News reporter, it said, "Michael Wooldridge, Professor of Artificial Intelligence," tick, "known for work on multi-agent systems," tick. And then it said it "studied at the University of Cambridge." And I never studied at the University of Cambridge. I never had any affiliation with the institute. So why is it getting it wrong? Because actually for an Oxford professor, that's quite a common background. And in the absence of any information, it's kind of filling in the gap in a very plausible way. But if somebody read that, they wouldn't notice that because it sounds plausible. So they get things wrong a lot and they get things wrong in extremely plausible ways. And that makes it very hard to detect those falsehoods. You may have heard recently that GPT-5 was released. So GPT-3 from OpenAI was really, for me anyway, the breakthrough large language model. That was really eye-opening. It was a huge step change in capability over its predecessor. And then GPT-4 released about two years after that was also a step change in capability. The one thing I think that would have made the biggest difference in GPT-5 was if it was better at not producing falsehoods. The problem is it still seems like that's a really, really big issue. And if I had to identify right now the single biggest barrier that's in the way of the wider take-up of these models, it's this fact that they just get things wrong so often. They get things wrong a lot and they get things wrong in very plausible ways.