IJBTT

Auto Tuning of Hadoop and Spark parameters

Auto Tuning of Hadoop and Spark parameters
	© 2021 by IJETT Journal
	Volume-69 Issue-11
	Year of Publication : 2021
	Authors : Mrs. Tanuja Patanshetti, Mr. Ashish Anil Pawar, Ms. Disha Patel, Mr. Sanket Thakare
	DOI :  10.14445/22315381/IJETT-V69I11P204

How to Cite?

Mrs. Tanuja Patanshetti, Mr. Ashish Anil Pawar, Ms. Disha Patel, Mr. Sanket Thakare, "Auto Tuning of Hadoop and Spark parameters," International Journal of Engineering Trends and Technology, vol. 69, no. 11, pp. 22-33, 2021. Crossref, https://doi.org/10.14445/22315381/IJETT-V69I11P204

Abstract
Data of the order of terabytes, petabytes, or beyond is known as Big Data. This data cannot be processed using the traditional database software, and hence there comes the need for Big Data Platforms. By combining the capabilities and features of various big data applications and utilities, Big Data Platforms form a single solution. It is a platform that helps to develop, deploy and manage the big data environment. Hadoop and Spark are the two open-source Big Data Platforms provided by Apache. Both these platforms have many configurational parameters, which can have unforeseen effects on the execution time, accuracy, etc. Manual tuning of these parameters can be tiresome, and hence automatic ways should be needed to tune them.
After studying and analyzing various previous works in automating the tuning of these parameters, this paper proposes two algorithms - Grid Search with Finer Tuning and Controlled Random Search. The performance indicator studied in this paper is Execution Time. These algorithms help to tune the parameters automatically. Experimental results have shown a reduction in execution time of about 70% and 50% for Hadoop and 81.19% and 77.77% for Spark by Grid Search with Finer Tuning and Controlled Random Search, respectively.

Keywords
Big Data Platform, Auto tuning, Parameters, Hadoop, Spark, Execution Time.

Reference
[1] https://hadoop.apache.org/
[2] https://spark.apache.org/
[3] Chin-Wei Hsu, Chih-Chung Chang and Chih-Jen Lin (2010). A practical guide to support vector classification. Technical Report, National Taiwan University.
[4] https://en.wikipedia.org/wiki/Hyperparameter_optimization
[5] Bergstra, James, and YoshuaBengio., Random search for hyper-parameter optimization., The Journal of Machine Learning Research 13(1) (2012) 281-305.
[6] Armanur Rahman, J.Hossen, Venkataseshaiah C., SMBSP: A Self-Tuning Approach using Machine Learning to Improve Performance of Spark in Big Data Processing., 7th International Conference on Computer and Communication Engineering (ICCCE) (2018).
[7] Wang, Ziyu, et al., Bayesian optimization in a billion dimensions via random embeddings., Journal of Artificial Intelligence Research 55 (2016) 361-387.
[8] W.L. Price., A Controlled Random Search Procedure for global optimization.
[9] Yigitbasi, Nezih, et al., Towards machine learning-based auto-tuning of mapreduce. 2013 IEEE 21st International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems. IEEE, (2013).
[10] Ding, Xiaoan, Yi Liu, and Depei Qian., Jellyfish: Online performance tuning with adaptive configuration and elastic container in Hadoop yarn., In 2015 IEEE 21st International Conference on Parallel and Distributed Systems (ICPADS), IEEE, (2015) 831-836.
[11] Bansal, Garvit, Anshul Gupta, Utkarsh Pyne, Manish Singhal, and Subhasis Banerjee., A framework for performance analysis and tuning in Hadoop-based clusters., In Smarter Planet and Big Data Analytics Workshop (SPBDA 2014), held in conjunction with International Conference on Distributed Computing and Networking (ICDCN 2014), Coimbatore, INDIA. (2014).
[12] Gu, Jing, Ying Li, Hongyan Tang, and Zhonghai Wu., Auto-Tuning Spark Configurations Based on Neural Network., In 2018 IEEE International Conference on Communications (ICC), (2018) 1-6. IEEE.
[13] Du, Haizhou, Ping Han, Wei Chen, Yi Wang, and Chenlu Zhang., Otterman: A Novel Approach of Spark Auto-tuning by a Hybrid Strategy., In 2018 5th International Conference on Systems and Informatics (ICSAI), IEEE, (2018) 478-483.
[14] Nguyen, Nhan, Mohammad Maifi Hasan Khan, and Kewen Wang., Towards automatic tuning of apache spark configuration., 2018 IEEE 11th International Conference on Cloud Computing (CLOUD). IEEE, (2018).
[15] Shivarkar, Sayali Ashok,. Speed-up extension to Hadoop system., International Journal of Engineering Trends and Technology (IJETT) (2014).