Big Data can create spectacular results. Retailers are using towerful technology to understand their customers more deeply and to grow their competitive advantage by manipulating information in new ways. Industrial enterprises are using it to streamline processes and logistics. Political candidates are using it to prioritize talking points and project poll results. The use cases are growing nearly as quickly as the data pools! But what, exactly, is making it so valuable?
“To maximize the investment in Big Data infrastructure, you must be able to produce conclusions that your rival cannot”
You’ve heard about the ‘Three V’s of Big Data’ – Volume, Variety and Velocity. These are the characteristics that personify Big Data, as opposed to, ‘regular data’. Volume - the sheer amount of data, measured in petabytes today and exabytes tomorrow. Variety - the ability to combine apples with oranges in quantifiable terms. Velocity - the return of query results in minutes rather than hours. We have taken huge steps forward in the amount of information that can be mined from the physical world and Big Data has grown up quickly to provide answers to the questions we never even dreamt before. Even four years ago, experts believed that we would produce 5,200 GB of data for every man, woman and child on Earth by 2020. That’s 40 zettabytes. Just how big is a zettabyte? IBM’s Jamie Thomas described it as “the contents of 20 million four-drawer file cabinets multiplied by a million.” Unbelievable! Today, we recognize that those estimates might be laughable, based upon the new sources and sensors available. The proliferation of data is simply jaw dropping.
Not all data is equal, however. More is not always better. Those ‘V’s’ don’t include “Value” and they don’t pragmatically help determine the difference between a Big Data repository that has extreme value compared to one that is borderline worthless. I propose that Big Data’s true value is ultimately based upon the combination of relevance and uniqueness.
Relevance, of course, based upon the fact that specificity can drive distinct queries and strategic advances. For example, a fishing company would have very little utility for voter demographics combined with the emotional ratings of Twitter comments about a reality show, no matter how much Volume, Variety or Velocity was produced! On the other hand, Comcast Cable’s collection of insights about their own customers’ viewing and spending habits would be extremely valuable. It would be extremely valuable to Charter Communications as well, in fact, which leads us to uniqueness.
To maximize the investment in Big Data infrastructure, you must be able to produce conclusions that your rival cannot. A few years ago, simply having the ability to crunch the numbers in this scale was rare enough to be the differentiator. By leveraging public information, even readily available government statistics, organizations could create an advantage just by using something better than a spreadsheet. Today, you must have proprietary data to append, or insight that allows you to ask better questions, or smarter personnel that know where to look for that edge. Ideally, you’d have all of the above, but even if you do not, you must be able to capitalize on the information that only you possess. This is the Moneyball paradox.
Michael Lewis’ 2003 best-selling novel, now a major motion picture starring Brad Pitt, chronicled the true story of Major League Baseball’s Oakland Athletics franchise and their efforts to find correlations and insight in the statistics where their competitors had overlooked it. Today, virtually all of the 30 professional baseball teams employ data specialists in an effort to eliminate that knowledge gap. As a result, the A’s have regressed from a perennial playoff contender, despite their relatively puny payroll, back to the middle of the pack. They lost their edge before they were able to capture a championship, because the rest of the league began applying decision sciences to those same statistical databases.
Compared to the Houston Astros and St. Louis Cardinals teams, which have been entangled in a federal investigation since June 2014 because of hacking claims, the Moneyball story seems rather provincial. The A’s were just using stats pulled from the public record. In contrast, a key employee left the Cardinals to take the top job in Houston, fueling his former colleagues’ paranoia about their proprietary information being improperly taken to the new employer. One thing led to another, and now a Cardinals employee will likely land in prison for hacking. Looking at the same data in a new way helped the Athletics in Moneyball, but it’s clearly the new data that is driving the competitive edge sought today by the Astros and the Cardinals. Their legal battle is a great example of the value placed upon the unique intel that each organization holds. They are willing to go to great lengths to protect it and so should you.
There is no point to investing time, money and effort into the ability to evaluate Big Data, if your best case scenario is arriving at the same conclusion as everyone else. It is the unique information that sheds the most light on the decision sciences – the information that your competitors wish that they had. It is the information that only you possess.. Whether it is cable company competitors or rival baseball teams, the proprietary internally harvested insights are the key differentiators and the reason that data security must go hand-in-hand with any Big Data project.
The possibility of your data being corrupted, stolen, replaced by random nonsense, or perhaps duplicated without your knowledge by a competitor is an unacceptable risk. Those V’s would be a lot less meaningful if your most valuable information was compromised, wouldn’t they? Volume would only be relevant when gauging just how much damage has been done. That’s a depressing thought.
As a result, vendors are racing against each other to build the latest and greatest tools to categorize, assess, and assign value to the data, minimizing risk for specific tranches of data that would be most destructive if lost. The intel is being safeguarded by military-grade encryption and employing user authentication to only grant access to verified personnel when needed. We are finally using technology to enforce a strict “need to know” policy, and not a moment too soon. We live in a world where the most lucrative of espionage is corporate. Regardless of your industry, Big Data’s value is derived from a single source – the quality of the data. If you have not considered the protection of this information a top priority, I hope you will now. It is imperative to the long-term survival of your organization.
