Data Analytics/BigPanda: Enabling Even More Effective Data Center Scale-Ups by Applying Data Science

Data centers have changed dramatically over the last decade.  In many cases, while the number of servers and overall complexity of the data center have grown dramatically, the size of the IT staff that’s managing the center hasn’t changed at all, or may even have shrunk.

When something goes wrong in a data center today, the time it takes to remedy IT problems can be extremely costly.  The average cost for a mid to large-size business system to  be down clocks in at $5600 every minute or $340,000 per hour.  Lost revenue from IT downtime is estimated to collectively hit businesses annually with a $26.5 billion price tag.

In order for a limited IT staff to be able to maintain a healthy operating environment, most data centers have deployed various monitoring tools that check for proper functioning of the servers, resources, network, and applications. The average data center today has at least five of these types of system monitoring tools running on their system.  These include tools like Nagios, Splunk, New Relic, Amazon CloudWatch, and Zenoss.  But a major problem is that none of these incident monitoring tools are compatible, and each produces different types of alerts and notifications, and each provides separate interfaces and languages for interacting with them.

So while most modern data centers would have a hard time functioning today without the use of incident monitoring tools, the tools themselves have become a bottleneck standing in the way to achieving and scaling for even greater data center efficiency.  For example, very often the effects of a single event or resource failure can cascade across various machines, resources and applications.  That cascade of events in turn triggers alerts which are generated and picked up by multiple monitoring tools.  To remedy the problem, IT staff needs to find the source of the problem and try to trace back through the history of alerts produced by the various monitoring tools.  This process of manually correlating alerts from the different tools can be time-consuming and error prone.

Assaf Resnick, Co-Founder and CEO of BigPanda, said that “you have now a scale of infrastructure that’s sending you thousands of alerts everyday, and you’re getting sent those alerts by several different systems, all of which speak a different language.  If you’re an IT admin, you’ll be sitting in a room looking at six different screens with thousands of alerts, and you have to manually  start understanding and investigating each of those alerts and figuring out, alright  from those thousands of alerts, what are the handful that are the most important and that are really affecting our customers and business.”

BigPanda is announcing today a cloud-based tool that simplifies incident management by aggregating alerts from all IT monitoring tools into a central repository and then applying data science algorithms to identify correlations between the alerts.  When patterns are identified, alerts can then be grouped into clusters and recognized as a single problem.  The BigPanda incident correlation and reduction algorithm also recognizes that a single event may be producing a noisy signature of many alerts and will match and combine all the alerts into a single cluster.  The data is then displayed visually in an easy to follow sequence of events on a problem timeline.  For the IT staff, this greatly simplifies the task of seeing exactly what the problem is and they can then concentrate on determining what the appropriate steps are to fix it.

Resnick said that “the new generation of IT infrastructure requires a fundamentally different approach to Incident Management. We believe that only through leveraging data science can IT teams tackle the scale of machines, events and dependencies that must be understood and managed. That’s why we founded BigPanda.”

Kevin Park, Head of Tech Ops and IT at Dropbox, said that “any modern Ops environment at scale will hit the pain-points BigPanda is solving. There’s a strong need for this product.”