A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters
Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y. 2016. A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3763
|Authors||Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y.|
At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment.
This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd.
|Journal||Concurrency and Computation: Practice and Experience|
|Journal citation||29 (3)|
|Digital Object Identifier (DOI)||https://doi.org/10.1002/cpe.3763|
|06 Jan 2016|
|Publication process dates|
|Deposited||03 Apr 2018|
1views this month
0downloads this month