Our data universe is expanding at a mind-blowing pace. The amount of data being generated from machines, phones, email, computers, etc. in the last two years alone surpasses the amount of data we’ve generated in the past. While enterprises have made significant investments in traditional BI tools, it’s a given that data warehousing and BI solutions are not agile and involves significant lag from specifying requirements to generating insights.
Big Data is a fairly overhyped term. Ultimately, analytics is about all forms of data. Big Data just deals with data sets of extremely high volumes and velocity that cannot be handled by traditional tools. That said, Big Data is enabling enterprises to change some of their fundamentals of how data is being analyzed altogether.
Big Data is helping enterprises rethink their constructs of information management. First, the scope of the data used for insights is changing from familiar structured data to all data i.e. structured and a variety of unstructured data. Second, as the landscape of this data grows, our focus is moving from defining the relationship between data entities to finding the correlation and interrelationship between data. Finally, it is becoming almost impossible to define the conditions to define these correlations. This means that the solutions that we develop to address Big Data challenges will need to be self-service in nature.
A very big change enterprises will have to deal with is to move away from the mentality of storing all their data in-house. The traditional method of analyzing data involved bringing all of it to a central location, break it down and analyze it. With the number and size of sources of data available now, there is a lot of focus on how enterprises are going to analyze these vast dispersed sources without physically bringing them together. This will require a very different breed of data platforms. Big Data solutions such as the Augmented Data warehouse leverage these technologies to help enterprises mine relevant information without having to invest heavily in storage and computing costs.
The plethora of data and relevant tools are clearly moving away from descriptive analytics to predictive analytics. For example, we were able to show one of the clients we work with, in the retail space, that store sales actually went up when the weather report indicated a coming storm. Traditionally, weather data would have never been a part of the analysis of store sales.
For another client, a high tech equipment manufacturer, we were able to understand the patterns of failures for their devices from their historical log data and apply these patterns in real time to predict failures before they happened. This solution was then extended to analyze the similarity of products as they were launched and predict the type of support that would be required for a new product even before it was launched by looking at the support patterns required for the other product.
We live in a world of nearly 7 billion connected devices that are generating loads of data every second. Leading organizations are already starting to leverage M2M (machine-to-machine) data and from the Internet of Things for product innovation, efficiency improvement, preventive maintenance and so on. Machine relevant data like this makes it possible to understand the utilization of infrastructure, workloads, business process, the risk and exposure of IT assets as well as their impact on core business processes, all of which is core to a CIO function.
Clearly, there is no doubt about the powerful analytics capabilities of Big Data. With the growing realization that all data needs to be scoped within a framework of holistic information management, enterprises look to tackle the challenge of “how’’ when it comes to using the various big data applications available to them.
This needs to be done through a single platform that allows us to seamlessly aggregate, access and analyze all this data within and outside the enterprise boundary. It should enable business and IT users alike. The goal is to deliver game changing insights that could be at the core of sustenance of the company in the market. Platforms like Big Data Hub at Infosys are a step in this direction.
