ETL, the process of transforming and loading data into a target system for analysis has traditionally been a long and tedious one, one that often resulted in lags of weeks or even months from when data was first collected until it reached a point where it could be analyzed.<\/div>\n
<\/div>\n
Now, newer systems are bringing improvements and efficiencies. \u00a0Increasingly data is being collected and moved in near real time to newer systems with Big Data processing capabilities, like Hadoop. \u00a0These Big Data systems act as a central data repository and can provide near real time analysis. \u00a0 While data still needs to be extracted from applications, very often data is first loaded into a repository like Hadoop and transformed there. \u00a0Rather than ETL, the new process is more like ELT, with transformation happening as the last step in the target system. \u00a0Some users of Hadoop, like Sears Holdings, are declaring that the ELT approach means “death to ETL<\/a>” and that it provides a solution to many of the problems and expense that ETL has caused.<\/div>\n
<\/div>\n
Phil Shelley<\/a>, CTO of Sears Holding and CEO of Metascale, said that<\/a> “The Holy Grail in data warehousing has always been to have all your data in one place so you can do big models on large data sets, but that hasn’t been feasible either economically or in terms of technical capabilities. \u00a0With Hadoop we can keep everything, which is crucial because we don’t want to archive or delete meaningful data… \u00a0ETL is an antiquated technique, and for large companies it’s inefficient and wasteful because you create multiple copies of data. \u00a0Everybody used ETL because they couldn’t put everything in one place, but that has changed with Hadoop, and now we copy data, as a matter of principle, only when we absolutely have to copy.”<\/div>\n
<\/div>\n
Newer technologies like Hadoop have certainly created efficiencies and brought improvements for how data analysis is done. \u00a0But the basic components of the ETL process still live on and really can’t be eliminated unless a business finds it possible to standardize on a single central repository for native use by all applications. \u00a0Without that, data still needs to be extracted from applications in order to move it into an analysis repository like Hadoop. \u00a0Once there, data still needs to be cleaned and appropriately transformed in order to process it. \u00a0And, at some point, whether before of after data transformation, the data needs to be loaded into the target system.<\/div>\n
<\/div>\n
James Markarian<\/a>, CTO of Informatica, pointed out<\/a> that whether you use newer Big Data technologies or continue to use Data Warehousing techniques, there are still problems that have to be addressed which just don’t go away. \u00a0Those problems include: \u00a0profiling data, discovering relationships between data, handling metadata, explaining context, accessing data, transforming data, cleansing data, and governing data for compliance.<\/div>\n![\"Digg](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/digg.png\" "\\"Digg")
<\/a><\/div>![\"Reddit](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/reddit.png\" "\\"Reddit")
<\/a><\/div>![\"Stumble](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/stumbleupon.png\" "\\"Stumble")
<\/a><\/div>![\"Buzz](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/yahoo_buzz.png\" "\\"Buzz")
<\/a><\/div>![\"Vote](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/dzone.png\" "\\"Vote")
<\/a><\/div>![\"Share](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/facebook.png\" "\\"Share")
<\/a><\/div>![\"Bookmark](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/delicious.png\" "\\"Bookmark")
<\/a><\/div>![\"Kick](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/dotnetkicks.png\" "\\"Kick")
<\/a><\/div>![\"Shout](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/dotnetshoutout.png\" "\\"Shout")
<\/a><\/div>![\"Share](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/linkedin.png\" "\\"Share")
<\/a><\/div>![\"Bookmark](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/technorati.png\" "\\"Bookmark")
<\/a><\/div>![\"Post](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/twitter.png\" "\\"Post")
<\/a><\/div>
<\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div><\/a><\/div>![\"Google](\"https:\/\/formtek.com\/blog\/wp-content\/plugins\/light-social\/google_buzz.png\" "\\"Google")
<\/a><\/div><\/div>","protected":false},"excerpt":{"rendered":"