Most of us are familiar with the idea of DevOps, a series of slick processes which combines applications development with IT Operations replacing the proven yet slower methods which IT teams used to build for enterprise applications.
It is fair to say DevOps is now becoming a ‘business as usual’ at most large organizations. Companies like Facebook and Google would not build apps in any other way. Even IT traditionalists, such as Gartner are cautiously optimistic about the new way of working, noting: ‘DevOps from here on will evolve from a niche strategy to a mainstream one employed by almost 25 per cent of Global 2000 organizations’.
Speed or ‘Time to Value’ and continuous improvements have made applications teams fall in love with DevOps making it a no-brainer for most. DevOps has redefined the software lifecycle and changed lives if not certainly IT careers. Now though it has a rebellious younger sister, DataOps, who promises the same improvements in terms of Time to Value with a view to continuously improve, not develop. But, who is this new kid on the block?
Some DataOps definitions may sound like an impossible dream, including ‘A data management method that emphasizes communication, collaboration, integration, automation and measurement of cooperation between data engineers, data scientists and other data professionals’. The reality is much less long-winded. We believe there are four layers to the DataOps process, which in most cases means four skill sets, if not four individuals, are involved in a DataOps cycle. As with DevOps, they involve both IT and non-IT professionals. From the top of the chain, they are:
• Information Consumers – makes business decisions e.g. CFOs, CIOs etc
• Business Analysts – understands the data and creates business information via data models e.g. Business Intelligence/Business Warehouse consultants, report writers and now Data Scientists.
• Platform Operations – understands how to manage the data platform e.g. Database Administrators and now DataOps professionals.
• Infrastructure Operations – understands how to manage the IT Infrastructure housing of the data e.g. managers of the hardware and Operating Systems, System Administrators, and now DataOps.
In our view, DataOps is particularly important if you have rolled out business intelligence (BI) products like SAP HANA and BW, which rely on effective collaboration between teams to make the best use of the data. It is this blending of teams, or skill sets, which make DataOps sound similar to DevOps. There are however some significant differences.
Large datasets and the real-time nature of modern data and intensive applications require some specific ‘Big Data’ skills. In addition, the business’s swelling ranks of information, consumers are demanding combined intelligence from the Internet of Things devices or vehicles and public data sources, to use the unstructured data and deliver real-time results from in-memory data. The term data scientist has started to take hold. This is where DevOps and DataOps diverge.
Along with the new job descriptions and the changing demands of the job, the analytical tools have become more complex. Instead of ‘merely’ blending the role of DBAs and IT operations professionals, the key skills required from the DataOps team are even more varied.
Amongst other things, DataOps gurus require a detailed understanding of the processes. The multi-tiered infrastructure is also required to extract business value from systems like HANA. Some of the questions which needed to be answered by the professionals in the near future may include:
• Where do I store the data needed to achieve results on time?
• How will I manage data growth and data retention to stay within budget?
• How do I report the cost of valuable data vs ‘dark data’?
• During a systems refresh, how do I manage test data sets so they return indicative results which approximate to the live production data set?
Given the scope of DataOps, it may be tempting to stick to ‘Business As Usual’ and in our view, this is not an option. If an organization is already using Business Intelligence, this will normally drive appetite for DataOps, because today’s Information Consumers are hungry to learn more and will demand more value from the data which is coursing through their organizations.
The growth of DataOps promises many of the benefits which DevOps is delivering from continuous improvement, to fasten Time to Value from IT initiatives. Its scope though is wider and will require a broader range of skill sets and touch, even more, individuals within an organization, perhaps even realigning departmental responsibilities.
When correctly deployed, DataOps ensures all the data is used to its full capacity by extracting the ‘goodness’ from every data interaction. Over time as organizations will realize its power and they will develop new frameworks which will be much more efficient in their extraction of intelligence from raw data. At Centiq, we are all in with DataOps and we think most organizations will double up both DevOps and DataOps professionals amongst their teams much sooner than IT Traditionalists.
