PERFORMANCE TUNING FOR EFFECTIVE SQOOP TO TRANSFORM BETWEEN SQL & NOSQL

Abstract

Transferring data to and from relational databases is challenging and laborious. Because data transfer requires careful handling, Apache Sqoop, short for “SQL to Hadoop,” was created to perform bidirectional data transfer between Hadoop and almost any external structured data store. Taking advantage of MapReduce, Hadoop’s execution engine, Sqoop performs the transfers in a parallel manner. It is very challenging task to transfer data with maximum performance and efficient manner. The variety of data sources and analytic targets presents a challenge in setting up effective data transfer pipelines. Data sources can have a variety of subtle inconsistencies: different DBMS providers may use different dialects of SQL, treat data types differently, or use distinct techniques to offer optimal transfer speeds. Depending on whether you’re importing to Hive, Pig, Impala, or your own MapReduce pipeline, you may want to use a different file format or compression algorithm when writing data to HDFS. Sqoop helps the data engineer tasked with scripting such transfers by providing a compact but powerful tool that flexibly negotiates the boundaries between these systems and their data layouts. In order to enhance the performance of Sqoop import and export operation, different parameters are configured in this project. Basic information regarding Sqoop import and export tool are presented in the different chapter. Analysis of configured Sqoop parameters is documented in the result and analysis chapter.