There is an unmistakable buzz in the air about Big Data, and it is similar, if not stronger than the dot com buzz. Big Data skill is the new H1B visa guarantee! At its heart, big data is about making immediate sense of huge volumes of structures and unstructured data moving at a tremendous pace – for instance, telecom usage data, or banking transactions data combined with the social and conversational data.
Yet, the unhealthy symptoms are unmistakable too! Most organizations do not have a business case for the use of Big Data. And invariably, a not so conducive culture to leverage big data technologies. This might just make Big Data the Big Fad of this decade. However, Big Data is a lot more than that.
What is driving the recent buzz?
CIOs and business leaders are drooling at the possibilities analysing so much information in real time can offer. Social media trends and patterns are creating newer ways of customer engagement. Global technology media is presenting multiple use cases in defence of big data, even as a few nay-sayers are talking about this being the peak of the big data hype cycle.
However, the focus in India seems to be – Real Time Reporting on Transaction Systems. Through the partitioning/clustering, distributed computing and in-memory processing methods, can big data run high quality real time reports, with the ability to drive additional queries? Yes, it can. But it will be not be the best use case for Big Data.
If we focus on this use case, IT will just end up being a cost centre. The new age IT organization needs to reinvent itself as a revenue driver, and globally, the trend is visible. Most CIOs and CTOs are emerging as a core piece of growth strategy.
Big Data investments should address either the fraud prevention story that helps the bottomline, or the customer offer story that helps the topline of a company. In both scenarios, big data solution delivers the ability to creatively process huge volume of stock (static) and flow (dynamic) information which may be structured or unstructured to make recommendations.
But, to get there, the immediate need is to get the data fixed. Data quality continues to remain a huge challenge across more than 80 percent of the mid and large organizations– the quality, integrity, consistency and accuracy of the data flowing through the source systems is often suspect, despite significant investments already. Reasons – the lack of a unified multi-channel architecture, the view of IT as a cost center, the weaker tie in of technology investments with a business case, the lack of scalability and ubiquity as one of the goals of technology investments, and the lack of will and funding to fix what is wrong.
If your stock data is suspect, and your flow data cannot be reliably linked to the stock, the mere possession of processing capability and intelligent algorithms will not solve all your problems. Also, the need to manage false positives in the real time is also high.
What this precipitates is the need to get the small/large data analytics in-place first. It is one of the most important questions today. What is the quality of insight generation team and infrastructure does the organization currently have? The answer cannot be – I have a team of statisticians and modelers, and that I have XYZ tool deployed. The real question is- what part of your forward looking strategy is driven on the back of your own core data?
And this is where data scientists… no, data artists will have a major role to play. For years, we have seen that successfully managing a business is as much an art as it is a science.
Today, at the intersection of business, technology and data, organizations need a data artist – someone who has the skills to converse in all three languages, and weave a symphony. Someone who does not mind breaking the mold a bit, and reimagining the future with the help of the tools the trade has to offer. Someone who’s not too fixated on the science of data, but the stories that data tells.
CIOs need to drive to this culture, and make the right investments, not just big investments.
