What is big data?

This week, we're talking about big data. But what actually is big data? In 2013, when he was the UK's science minister, Lord David Willetts hailed it as one of his 8 great technologies - industries in which the UK excels. He explains why.

David -   Big data is using the power of high performance computing, plus very smart software to detect patterns in large data sets that aren't always apparent using conventional statistical techniques.  More data has been generated in the past five years than in all previous human history. I identified it as one of the eight great technologies because Britain has got some distinctive strengths.  The fact is we've got some pretty powerful computers, but not absolutely top five computers. 

The real way to get value from these computers is to write smart software that gets more results with fewer calculations, and we do write very good software. 

We're good at extracting significance from computers even if they're not always the most powerful. 

And then what really matters for us in Britain, is as well as all the contemporary flow of data, we've also in Britain got some really long historic data sets. 

Reliable records kept for much longer periods than most other countries, from measuring climate change by tracking weather records kept by vicars in their parsonage.  We have got some amazing resource like that which does reveal patterns that otherwise would not have been spotted.