Big data: Could the end really be in sight?
I can't say I love the term Big Data. How big is big? When it comes to data, often what was big last year is just the norm this year.
What I am sure about is that Big Data is Massive now. Big just doesn't do it justice. The great thing for data is that it has a perfect mate in the "Internet of Things". Once IoT and data get their act together we'll really start to appreciate just how big it can get. When everything we use is connected and interacting, data is going to increase exponentially, and we'll need some new hyperbolic terminology to describe it.
The thing that has struck me recently is just how the NZ business community is only just beginning to scratch the surface of managing their data. I was pleased to see how our utilities, banking and FMCG industries are basing their growth around data management. I was even more pleased that our healthcare providers, particularly private hospitals, are using analytics to improve their services. The fact that both these industries are really in their infancy when it comes to using their data to drive their business decisions just shows how much there is to go.
The thing about Big Data is that, Big is not necessarily good. If you don't use a fraction of that data you have access to then you are just overpaying for storage space. Increasingly data will be seen as strategically important for medium and small sized enterprises as they seek insights into the huge amount of data they keep, it's what they do with it that matters.
Traditional reporting tends to be retrospective, however analytics has a predictive focus which requires more sophisticated data modelling and analysis tools. Good news is that the tools for this are becoming simpler all the time and will eventually take the analysis away from the IT team and into the hands of the business.
Fewer reports will be on spreadsheets and more will be in easily interpreted graphical formats using data visualisation tools to highlight patterns and trends. When it comes to a business getting the competitive edge it's essential that results are easily interpreted by the business leaders.
Clean, Fast and Useful data will become more important than Big. That is easier said than done as data is notoriously tricky. Apart from taking various guises or formats it also lives in different places and even hides in the cloud. Yet, as data usage grows, so does our expectations of visualisation and insights. At some stage in the not too distant future machine learning and automated algorithms will allow us to order and process our data. For now however, we'll have to remain manual with Datawarehousing and Business Intelligence Consultants providing the hard work behind a data driven business culture. It's likely that companies will start to look towards a Chief Data Officer to guide us through this business transformation.
What is pleasing is that most businesses I talk to; using data to grow their business, do so from the perspective of customer centricity – by providing a service or product better to the customer than their competitors. As the IoT becomes a reality, where every device or appliance we use reports back something about us, the potential for services to be tailored specifically for an individual is enormous. I'm happy that my doctor will predict when I need a health check up based on my eating habits and little of consequence coming back from my fitness app. Of course this is also a scary prospect. Soon there will be nowhere to hide and there's a whole new discussion to be had on privacy and security.
So, yes, this is the end for Big Data. When it's this big I don't think we need it pointed out anymore.
