A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters
Journal article
Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y. 2016. A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3763
Authors | Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y. |
---|---|
Abstract | At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment. This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd. |
Year | 2016 |
Journal | Concurrency and Computation: Practice and Experience |
Journal citation | 29 (3) |
Publisher | Wiley |
ISSN | 1532-0626 |
Digital Object Identifier (DOI) | https://doi.org/10.1002/cpe.3763 |
Publication dates | |
06 Jan 2016 | |
Publication process dates | |
Deposited | 03 Apr 2018 |
Output status | Published |
https://repository.canterbury.ac.uk/item/88q19/a-sliding-window-based-dynamic-load-balancing-for-heterogeneous-hadoop-clusters
46
total views0
total downloads1
views this month0
downloads this month
Export as
Related outputs
Repairing process models with non-free-choice constructs based on token replay
Bai, E., Qi, M., Luan, W., Li, P. and Du, Y. 2022. Repairing process models with non-free-choice constructs based on token replay. Computing and Informatics. 41 (4), pp. 1054-1077. https://doi.org/10.31577/cai_2022_4_1054Transformers only look once with nonlinear combination for real-time object detection
Xia, R., Li, G., Huang, Z., Pang, Y. and Qi, M. 2022. Transformers only look once with nonlinear combination for real-time object detection. Neural Computing and Applications. https://doi.org/10.1007/s00521-022-07333-yDeviation detection in clinical pathways based on business alignment
Tian, Y., Li, X., Qi, Man, Han, D. and Du, Yuyue 2022. Deviation detection in clinical pathways based on business alignment. Scientific Programming. 2022, pp. 1-13. https://doi.org/10.1155/2022/6993449Security vulnerabilities of popular smart home appliances
Qi, M., Induruwa, A. and Hussain, F. 2021. Security vulnerabilities of popular smart home appliances. in: Proceeding of The Twentieth International Conference on Networks April 18, 2021 to April 22, 2021 - Porto, PortugalImproved adaptive genetic algorithm for the vehicle insurance fraud identification model based on a BP neural network
Yan, C., Li, M., Liu, W. and Qi, M. 2020. Improved adaptive genetic algorithm for the vehicle insurance fraud identification model based on a BP neural network. Theoretical Computer Science. 817, pp. 12-23. https://doi.org/10.1016/j.tcs.2019.06.025Hybrid Intrusion Detection System for Smart Home Applications
Hussain, Fida, Induruwa, Abhaya and Qi, Man 2020. Hybrid Intrusion Detection System for Smart Home Applications. in: Mahmood, Z. (ed.) Developing and Monitoring Smart Environments for Intelligent Cities IGI Global. pp. 300-322Payments per claim model of outstanding claims reserve based on fuzzy linear regression
Yan, C., Liu, Q., Liu, J., Liu, W., Li, M. and Qi, M. 2019. Payments per claim model of outstanding claims reserve based on fuzzy linear regression. International Journal of Fuzzy Systems. 21, pp. 1950-1960. https://doi.org/10.1007/s40815-019-00617-xFuzzy interacting multiple model H∞ particle filter algorithm based on current statistical model
Wang, Q., Chen, X., Zhang, L., Li, J., Zhao, C. and Qi, M. 2019. Fuzzy interacting multiple model H∞ particle filter algorithm based on current statistical model. International Journal of Fuzzy Systems. 21, pp. 1894-1905. https://doi.org/10.1007/s40815-019-00678-yTemporal sparse feature auto-combination deep network for video action recognition
Wang, Q., Gong, D., Qi, M., Shen, Y. and Lei, Y. 2018. Temporal sparse feature auto-combination deep network for video action recognition. Concurrency and Computation: Practice and Experience. https://doi.org/10.1002/cpe.4487Soundness analytics of composed logical workflow nets
Liu, W., Wang, L., Feng, X., Qi, M., Yan, C. and Li, M. 2017. Soundness analytics of composed logical workflow nets. International Journal of Parallel Programming. https://doi.org/10.1007/s10766-017-0536-8Gaussian-Gamma collaborative filtering: a hierarchical Bayesian model for recommender systems
Luo, C., Zhang, B., Xiang, Y. and Qi, M. 2017. Gaussian-Gamma collaborative filtering: a hierarchical Bayesian model for recommender systems. Journal of Computer and System Sciences. https://doi.org/10.1016/j.jcss.2017.03.007Facilitating visual surveillance with motion detections
Qi, M. 2017. Facilitating visual surveillance with motion detections. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3770Data security of android applications
Obiri-Yeboah, J. and Qi, M. 2016. Data security of android applications. in: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery : ICNC-FSKD 2016 : 13-15 August, Changsha, China IEEE Xplore.AL-DDCNN : a distributed crossing semantic gap learning for person re-identification
Cheng, K., Zhan, Y. and Qi, M. 2017. AL-DDCNN : a distributed crossing semantic gap learning for person re-identification. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3766