Big Data: The Ground Hog Cycle for In-Memory Computing

In Gartner’s list of top-10 strategic technologies for 2012, it lists In-Memory Computing at number eight.  Just what is that, and what’s the benefit of using it?

Quite simply it is adding large amounts of available memory to computers to speed up calculations of all types.  Solving problems exclusively in memory, rather than relying on the hard drive to cache data, can dramatically improve the time to perform complex computer calculations.  SAP’s HANA (High Performance Analytics Appliance) in-memory system is said to be 3600 times faster than a computer similarly configured but without the large amounts of memory. That means that calculations that may have taken hours can now be done in seconds.

SAP has jumped into in-memory computing in a big way, and SAP’s Chief Scientist Ike Nassi says that there’s nothing new about In-Memory Computing, it’s simply a rebirth of an idea dating back twenty or more years.  He likens the rebirth to a sort of Ground Hog Day effect, referring to the popular movie starring Bill Murray about a weatherman who was caught up in a time-warp that doomed him to repeat his life on Ground Hogs’ Day over and over.  Trends in technology repeat themselves.  Nassi compares Big Data problems of today with speed issues of Lotus-123 spreadsheets back in the ’80’s.  Back then, spreadsheets only managed small amounts of data, but even though data sizes were small, the data still couldn’t all fit into computer memory, and calculations were slow.  Things speeded up significantly though as computers were gradually armed with more and more megabytes of fast DRAM chips.

Similar to the kick start in application development and sales that DRAM gave computers in the ’80’s, in-mmeory computing is likely to continue to push the boundaries on what is possible for application development.  Expect a new category of data-based applications to emerge.

Since today is Ground Hogs’ Day, it seemed only proper to talk about in-memory computing.  Today, the promise of Big Data is driving strong interest in using computers decked out with ever larger amounts of fast computer memory.  Because of Big Data, another cycle of adding large amounts of memory to computers is under way.

And, it’s not just SAP that is promoting In-Memory Computing.  Products using in-memory computing already today include SAS (High Performance Computing Package), Microsoft’s PowerPivot and 64-bit SQL Server database “Denali“, the Open Source Pentaho data analytics package all offer in-memory computing options.  Expect more vendors and products to follow  the trend.

Alan Bowling on ComputerWeekly writes that “For certain sectors where huge amounts of data are practically a requirement, such as utilities or finance, in-memory computing is already a hugely disruptive technology. Organisations in these sectors would do well to make the move to in-memory computing early, rather than being left trying to catch up with the competition.”