Over the past decade, devops has become globally recognized and accepted as necessary for high-performing IT organizations. There have been countless examinations of the role of devops, all concluding that business health cannot be maintained without it.
By using either Cloud infrastructure or self-hosting, companies are making sure that machines, virtual systems, and services are constantly monitored. Data collectors are installed to ensure thorough alerts and dashboards, servers and services are constantly monitored for continuity of the system. Still, devops isn’t foolproof, in fact I can cite hundreds of major and expensive incidents that even devops couldn’t protect businesses from facing. Fortunately, in the past year or two, there is a better understanding that devops is only one part of the puzzle that keeps businesses healthy.
WHAT’S CHANGING IN DEVOPS?
One of the main reasons that costly incidents still occur is because as good at monitoring as devops is, it is entangled with BI and both have what the other needs. Devops understands monitoring without a holistic understanding of the business and its granular data. On the other end of the spectrum are BI and data teams that do have a nuanced understanding of business data, but are lacking in tools for around-the-clock monitoring and alerting to abnormal behavior of the data.
This schism opens businesses to many vulnerabilities; from buggy software to bad integration or external factors like new behavior by a competitor or integration changes, all of which can affect a business and its operation. As data production grows exponentially businesses are seeing new metrics which they were never aware of, again opening the door to potential issues and revealing a fuller and more accurate picture of the business’s status. This is well and good but without attention in the right place and dedicated monitoring role this new knowledge and bigger picture cannot deliver its expected benefit.
MINDING THE GAP WITH DATAOPS
To handle the misleading perception that companies can be on their monitoring A-game with devops alone, an entirely new role is needed: dataops. Because of the dynamic nature of data and the constant new services, partnerships, and products entering the market every quarter, the dataops role is ongoing and should comprehensively understand and use the proper tools to monitor the ebb and flow of company data including business anomalies, trend changes, changes in predictions, etc.
This role should be aware of all company data, understand its flow and behavior, and be alert to incidents like major data leaks, broken data, data integrity issues, and their context (drops in revenue, visitor, usage, checkout failure, payment/transaction issues, etc.). The methodology to do the job should remain the same as devops – 24/7 monitoring with instant alerts in case of a critical incident. It is a well-known fact that companies lose millions of dollars due to glitches, incidents, and other failures which are entirely preventable; the potential saved revenue alone justifies the existence of a dataops position.
Others share this opinion that dataops is the wave of the future. In his May 2017 report, “No More Silos: How DataOps Technologies Overcome Enterprise Data Isolationism,” Toph Whitmore of Blue Hill Research writes: “Enterprises that deploy DataOps models to establish the free flow of data within their organizations will see new efficiencies, realize new insights, accelerate time to action, and maximize data-derived value.”
BI IS NOT DATAOPS
From experience, companies use BI to review and analyze company data, but it was never a 24/7 job, and as we know, data never sleeps. Despite knowing this, most business intelligence professionals still haven’t developed the “network operations center-type” always-on mentality. Only a new role with a new name and a clear mission will keep dataops as the focus of continuous business monitoring.
As the focus of dataops is to monitor and understand all company data, there is a strong existing link between this role and existing company roles like BI analysts and data engineers. Each role is unique enough to stand on its own, and all three should be reporting to a Chief Data Officer, a position that is becoming increasingly prevalent in data-driven companies.
To introduce new roles to entrenched companies is not an easy thing, but adoption of new practices never is. One thing is certain though – to keep up with data, businesses are going to have to reframe the way they think about monitoring and develop new job roles to accommodate for this change.
By David Drai CEO and cofounder of Anodot
