Mike Lynch: Big data may be big but is it clever?

The biggest change the IT industry is facing is that machines are getting cleverer, and that means moving beyond the limitations of the traditional database, according to Mike Lynch, entrepreneur and former CEO of British software firm Autonomy.

Speaking at the IT Expo in London today Lynch said that, in the 1960s, people realised that computers could be used to carry out tasks, but that the real world was far too complicated for machines to deal with. They therefore looked for easily defined tasks that could be turned into structured information, and went on to build relational databases.

"The IT industry was born, and we fired some people who did lots of adding up," he said.

Since then, IT companies have simply been creating updated versions of those databases and calling them new names, in order to sustain industry growth. The relational database, the data warehouse, business intelligence and analytics are all just variations on a theme, said Lynch.

"Now we're going to sell you a database that's bigger than the others and that's called Big Data," said Lynch.

"The important thing about this is, it's big but is it clever? Actually the real frontier in big data is the type of information that the databases haven't been able to understand - the vast 95% of information out there which is email, text, video and audio, because that's where the real competitive value comes."

Lynch said that most organisations do not have structured data on the scale that they need true big data technologies. However, most organisations do have unstructured or "human-friendly" data on that scale.

"The ability for a computer now to read an email, listen to a phone call or watch a piece of TV and understand what it means, means we're hitting a brave new world of automation," he said.

Lynch pointed to the Internet of Things as an indication of the vast amounts of data that will need to be dealt with and understood and fused together.

"We've all heard about our fridge telling us about our milk - this is happening," he said. "That means that data has got to be understood at a meaningful level, not just does field A equal field B?"

Lynch concluded that IT is far more than a database app, and that the IT part of the organisation has become more and more important in creating a differentiation.

"In the Internet of Things, there will be winners and losers, but it will be those companies that own the data who will find themselves in the best position in future," he said.