A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters

Journal article

Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y. 2016. A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3763
AuthorsLiu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y.
Abstract

At present MapReduce computing model‐based Hadoop framework has gradually become the most famous distributed computing framework because of its remarkable features such as scalability, fault tolerance, data security, and powerful IO ability. However, Hadoop framework only supports limited load balancing policies, which may result in performance deterioration in heterogeneous clusters. Additionally Hadoop does not have advanced dynamic load balancing mechanism in enabling its optimal performance in dynamic environment.

This paper presents a sliding window‐based dynamic load balancing algorithm, which specially aims at balancing the load among the heterogeneous nodes during the Hadoop job processing. The presented algorithm is evaluated in both simulated and physical environments. The experimental results show that the performances in terms of efficiency of Hadoop cluster can be significantly improved. Copyright © 2016 John Wiley & Sons, Ltd.

Year2016
JournalConcurrency and Computation: Practice and Experience
Journal citation29 (3)
PublisherWiley
ISSN1532-0626
Digital Object Identifier (DOI)https://doi.org/10.1002/cpe.3763
Publication dates
Print06 Jan 2016
Publication process dates
Deposited03 Apr 2018
Output statusPublished
Permalink -

https://repository.canterbury.ac.uk/item/88q19/a-sliding-window-based-dynamic-load-balancing-for-heterogeneous-hadoop-clusters

  • 81
    total views
  • 0
    total downloads
  • 1
    views this month
  • 0
    downloads this month

Export as

Related outputs

Focusing on the golden skills of effective communication and collaboration to further enhance graduate employability attributes
O'Leary, S., McGee, M., Qi, M., Millns, S. and Eyden, A. 2024. Focusing on the golden skills of effective communication and collaboration to further enhance graduate employability attributes. Graduate College Working Papers.
Feed-forward ANN with Random Forest Technique for Identifying malicious Internet of Things network intrusion
Obarafor, Victor, Qi, Mandi and Zhang, Leishi 2024. Feed-forward ANN with Random Forest Technique for Identifying malicious Internet of Things network intrusion. in: 2024 2nd International Conference on Cyber Resilience (ICCR) IEEE.
The role of use cases when adopting augmented reality into higher education pedagogy
Ward, G., Turner, S., Pitt, C., Qi, M., Richmond-Fuller, A. and Jackson, T. 2024. The role of use cases when adopting augmented reality into higher education pedagogy.
A review of privacy-preserving federated learning, deep learning, and machine learning IIoT and IoTs solutions
Obarafor, Victor, Qi, Man and Zhang, L. 2023. A review of privacy-preserving federated learning, deep learning, and machine learning IIoT and IoTs solutions. in: 2023 8th IEEE International Conference on Signal and Image Processing (ICSIP) Wuxi, China IEEE. pp. 1074-1078
Pre-processing of social media remarks for forensics
Gao, Xuhao and Qi, Man 2023. Pre-processing of social media remarks for forensics. IEEE. https://doi.org/10.1109/icnc-fskd59587.2023.10280980
Alignment-based conformance checking of hierarchical process models
Wang, L, Han, X., Qi, M., Wang, K. and Lu, P. 2023. Alignment-based conformance checking of hierarchical process models. Computing and Informatics. 43 (1), pp. 149-180. https://doi.org/10.31577/cai_2024_1_149
Research on UBI auto insurance pricing model based on parameter adaptive SAPSO optimal fuzzy controller
Wang, X., Liu, W., Ou, Z., Yan, C., Qi, M. and Jia, W. 2022. Research on UBI auto insurance pricing model based on parameter adaptive SAPSO optimal fuzzy controller. Computing and Informatics. 41 (4), pp. 1078-1113. https://doi.org/10.31577/cai_2022_4_1078
Novel architecture for human re-identification with a two-stream neural network and attention ,echanism
Rahi, B. and Qi, M. 2022. Novel architecture for human re-identification with a two-stream neural network and attention ,echanism. Computing and Informatics. 41 (4), pp. 905-930. https://doi.org/10.31577/cai_2022_4_905
A novel data analytic model for mining user insurance demands from microblogs
Chun Yan, Lu Liu, Wei Liu and Man Qi 2022. A novel data analytic model for mining user insurance demands from microblogs. Computing and Informatics. 41 (3). https://doi.org/10.31577/cai_2022_3_689
Deep learning based real-time facial mask detection and crowd monitoring
Chan-Yun Yang, Hooman Samani, Nana Ji, Chunxu Li, Ding-Bang Chen and Man Qi 2022. Deep learning based real-time facial mask detection and crowd monitoring. Computing and Informatics. 40 (6), pp. 1263-11294. https://doi.org/10.31577/cai_2021_6_1263
Repairing process models with non-free-choice constructs based on token replay
Bai, E., Qi, M., Luan, W., Li, P. and Du, Y. 2022. Repairing process models with non-free-choice constructs based on token replay. Computing and Informatics. 41 (4), pp. 1054-1077. https://doi.org/10.31577/cai_2022_4_1054
Transformers only look once with nonlinear combination for real-time object detection
Xia, R., Li, G., Huang, Z., Pang, Y. and Qi, M. 2022. Transformers only look once with nonlinear combination for real-time object detection. Neural Computing and Applications. https://doi.org/10.1007/s00521-022-07333-y
Deviation detection in clinical pathways based on business alignment
Tian, Y., Li, X., Qi, Man, Han, D. and Du, Yuyue 2022. Deviation detection in clinical pathways based on business alignment. Scientific Programming. 2022, pp. 1-13. https://doi.org/10.1155/2022/6993449
Security vulnerabilities of popular smart home appliances
Qi, M., Induruwa, A. and Hussain, F. 2021. Security vulnerabilities of popular smart home appliances. in: Proceeding of The Twentieth International Conference on Networks April 18, 2021 to April 22, 2021 - Porto, Portugal
Improved adaptive genetic algorithm for the vehicle insurance fraud identification model based on a BP neural network
Yan, C., Li, M., Liu, W. and Qi, M. 2020. Improved adaptive genetic algorithm for the vehicle insurance fraud identification model based on a BP neural network. Theoretical Computer Science. 817, pp. 12-23. https://doi.org/10.1016/j.tcs.2019.06.025
Hybrid Intrusion Detection System for Smart Home Applications
Hussain, Fida, Induruwa, Abhaya and Qi, Man 2020. Hybrid Intrusion Detection System for Smart Home Applications. in: Mahmood, Z. (ed.) Developing and Monitoring Smart Environments for Intelligent Cities IGI Global. pp. 300-322
Payments per claim model of outstanding claims reserve based on fuzzy linear regression
Yan, C., Liu, Q., Liu, J., Liu, W., Li, M. and Qi, M. 2019. Payments per claim model of outstanding claims reserve based on fuzzy linear regression. International Journal of Fuzzy Systems. 21, pp. 1950-1960. https://doi.org/10.1007/s40815-019-00617-x
Fuzzy interacting multiple model H∞ particle filter algorithm based on current statistical model
Wang, Q., Chen, X., Zhang, L., Li, J., Zhao, C. and Qi, M. 2019. Fuzzy interacting multiple model H∞ particle filter algorithm based on current statistical model. International Journal of Fuzzy Systems. 21, pp. 1894-1905. https://doi.org/10.1007/s40815-019-00678-y
Temporal sparse feature auto-combination deep network for video action recognition
Wang, Q., Gong, D., Qi, M., Shen, Y. and Lei, Y. 2018. Temporal sparse feature auto-combination deep network for video action recognition. Concurrency and Computation: Practice and Experience. https://doi.org/10.1002/cpe.4487
Soundness analytics of composed logical workflow nets
Liu, W., Wang, L., Feng, X., Qi, M., Yan, C. and Li, M. 2017. Soundness analytics of composed logical workflow nets. International Journal of Parallel Programming. https://doi.org/10.1007/s10766-017-0536-8
Gaussian-Gamma collaborative filtering: a hierarchical Bayesian model for recommender systems
Luo, C., Zhang, B., Xiang, Y. and Qi, M. 2017. Gaussian-Gamma collaborative filtering: a hierarchical Bayesian model for recommender systems. Journal of Computer and System Sciences. https://doi.org/10.1016/j.jcss.2017.03.007
Facilitating visual surveillance with motion detections
Qi, M. 2017. Facilitating visual surveillance with motion detections. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3770
Data security of android applications
Obiri-Yeboah, J. and Qi, M. 2016. Data security of android applications. in: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery : ICNC-FSKD 2016 : 13-15 August, Changsha, China IEEE Xplore.
AL-DDCNN : a distributed crossing semantic gap learning for person re-identification
Cheng, K., Zhan, Y. and Qi, M. 2017. AL-DDCNN : a distributed crossing semantic gap learning for person re-identification. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3766
Integration of link and semantic relations for information recommendation
Zhao, Q., He, Y., Wang, P., Jiang, C. and Qi, M. 2016. Integration of link and semantic relations for information recommendation. Computing and Informatics. 35 (1), pp. 30-54.
Data-driven pedestrian re-identification based on hierarchical semantic representation
Cheng, K., Xu, F., Tao, F., Qi, M. and Li, M. 2018. Data-driven pedestrian re-identification based on hierarchical semantic representation. Concurrency and Computation: Practice and Experience. 30 (23). https://doi.org/10.1002/cpe.4403
Integrated privacy preserving framework for smart home
Qi, M. and Hussain, F. 2018. Integrated privacy preserving framework for smart home. in: Proceedings of The 14th International Conference on Fuzzy Systems and Knowledge Discovery, 28-30 July 2018 in Huangshan, China.
Parallelizing abnormal event detection in crowded scenes with GPU
Yavari, M., Li, M., Li, S. and Qi, M. 2015. Parallelizing abnormal event detection in crowded scenes with GPU.
Big data management in digital forensics
Qi, M. 2014. Big data management in digital forensics. in: 2014 IEEE 17th International Conference on Computational Science and Engineering (CSE) IEEE. pp. 238-243
Digital forensics and NoSQL databases
Qi, M. 2014. Digital forensics and NoSQL databases. in: Proceedings of the 11th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), 2014 IEEE. pp. 734-739
Employing neural networks for DDoS detection
Qi, M. 2014. Employing neural networks for DDoS detection.
A MapReduce based distributed LSI for scalable information retrieval
Liu, Y., Li, M., Khan, M. and Qi, M. 2014. A MapReduce based distributed LSI for scalable information retrieval. Computing and Informatics. 33 (2), pp. 259-280.
Clear up grey areas: cybercrime legislation progress in China
Qi, M. 2013. Clear up grey areas: cybercrime legislation progress in China.
A resource aware MapReduce based parallel SVM for large scale image classification
Guo, W., Khalid, N., Liu, Y., Li, M., Qi, M., Guo, W., Khalid, N., Liu, Y., Li, M. and Qi, M. 2015. A resource aware MapReduce based parallel SVM for large scale image classification. Neural Processing Letters. 44 (1), pp. 161-184.
Social media in law enforcement: the role and issues
Qi, M. 2012. Social media in law enforcement: the role and issues.
A distributed storage system for archiving broadcast media content
Cherry, D., Li, M. and Qi, M. 2012. A distributed storage system for archiving broadcast media content. in: Grid and Cloud Computing: Concepts, Methodologies, Tools and Applications IGI Global. pp. 669-679
A distributed SVM ensemble for image classification and annotation
Alham, M., Li, M., Liu, Y., Ponraj, M. and Qi, M. 2012. A distributed SVM ensemble for image classification and annotation. in: 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) IEEE. pp. 1581-1584
Computer forensics in communication networks
Wang, Y. and Qi, M. 2012. Computer forensics in communication networks. in: IET International Communication Conference on Wireless Mobile and Computing (CCWMC 2011) Curran.
A MapReduce based parallel SVM for large scale spam filtering
Caruana, G., Li, M. and Qi, M. 2011. A MapReduce based parallel SVM for large scale spam filtering. in: Caruana, G. (ed.) Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), 2011 IEEE.
Cyber safety in schools
Qi, M. 2011. Cyber safety in schools.
The social effects of spam
Mousoli, R., Qi, M. and Edgar-Nevill, D. 2008. The social effects of spam. in: Edgar-Nevill, D. (ed.) Proceedings of CFET 2008 – 2nd Annual International Conference on Cybercrime Forensics Education and Training Canterbury Canterbury Christ Church University. pp. 71-78
A distributed SVM ensemble for large scale image classification and annotation
Alham, K., Li, M., Liu, Y. and Qi, M. 2013. A distributed SVM ensemble for large scale image classification and annotation. Computers and Mathematics with Applications. 66 (10), pp. 1920-1934. https://doi.org/10.1016/j.camwa.2013.07.015
Fast implementation of cel-based animation
Qi, M. 2008. Fast implementation of cel-based animation.
A case-based reasoning model for digital intrusion forensics
Qi, M., Wang, Y. and Xu, R. 2008. A case-based reasoning model for digital intrusion forensics.
Cybercrime legislation in China
Qi, M., Wang, Y. and Xu, R. 2008. Cybercrime legislation in China.
P2P DDoS: Challenges and Countermeasures
Qi, M. 2009. P2P DDoS: Challenges and Countermeasures.
Distributed indexing for resource discovery in P2P(Peer to Peer) networks
Qi, M., Hentschel, M., Li, M. and Ponraj, M. 2009. Distributed indexing for resource discovery in P2P(Peer to Peer) networks.
P2P(peer to peer) network-targeted DDoS(distributed denial-of-service attacks)
Qi, M. 2009. P2P(peer to peer) network-targeted DDoS(distributed denial-of-service attacks).
Spam and social effects
Qi, M., Edgar-Nevill, D. and Mousoli, R. 2009. Spam and social effects.
Footprints of cyber criminals
Qi, M. and Edgar-Nevill, D. 2007. Footprints of cyber criminals.
Tracking email offenders
Edgar-Nevill, D. and Qi, M. 2007. Tracking email offenders.
Protecting intellectual property and computer software: legislation in China
Qi, M. and Wang, Y. 2010. Protecting intellectual property and computer software: legislation in China.
Machine learning based spam filtering: advantages and challenges
Qi, M. 2010. Machine learning based spam filtering: advantages and challenges.
OBIRE: ontology based bibliographic information publication and retrieval
Liu, X., Li, M., Liu, Y. and Qi, M. 2010. OBIRE: ontology based bibliographic information publication and retrieval. International Journal of Distributed Systems and Technologies. 1 (4), pp. 58-73.
Cybercrime and relevant legislation in China
Qi, M. and Wang, Y. 2010. Cybercrime and relevant legislation in China. in: Reich, P. (ed.) Cybercrime and Security Oxford Oxford University Press. pp. 1-34
Semantic analysis for spam filtering
Qi, M. and Mousoli, R. 2010. Semantic analysis for spam filtering.
MAPBOT: A web based map information retrieval system
Li, M. and Qi, M. 2003. MAPBOT: A web based map information retrieval system. Information and Software Technology. 45 (10), pp. 691-698. https://doi.org/10.1016/S0950-5849(03)00082-X
Leveraging legacy codes to distributed problem-solving environments: a web services approach
Li, M. and Qi, M. 2004. Leveraging legacy codes to distributed problem-solving environments: a web services approach. Software: Practice and Experience. 34 (13), pp. 1297-1309. https://doi.org/10.1002/spe.614
PGGA: a predictable and grouped genetic algorithm for job scheduling
Li, M., Yu, B. and Qi, M. 2006. PGGA: a predictable and grouped genetic algorithm for job scheduling. Future Generation Computer Systems. 22 (5), pp. 588-599. https://doi.org/10.1016/j.future.2005.09.001
Lessons learned from Beijing for the London 2012
Edgar-Nevill, D. and Qi, M. 2009. Lessons learned from Beijing for the London 2012. in: Edgar-Nevill, D. (ed.) Proceedings of the 3rd International Conference on Cybercrime Forensics Euducation and Training CFET 2009 Canterbury, UK Canterbury Christ Church University.
Tracking online trails
Qi, M., Edgar-Nevill, D., Wang, Y. and Xu, R. 2008. Tracking online trails. in: Jahankhani, H., Revett, K. and Palmer-Brown, D. (ed.) Global E-Security Springer.
Web services discovery with rough sets
Li, M., Yu, B., Sahota, V. and Qi, M. 2009. Web services discovery with rough sets. International Journal of Web Services Research. 6 (1), pp. 69-86.
Facilitating resource discovery in grid environments with peer-to-peer structured tuple spaces
Li, M. and Qi, M. 2009. Facilitating resource discovery in grid environments with peer-to-peer structured tuple spaces. Peer-to-Peer Networking and Applications. 2 (4), pp. 283-297. https://doi.org/10.1007/s12083-009-0036-8
Optimizing peer selection in BitTorrent networks with genetic algorithms
Wu, T., Li, M. and Qi, M. 2010. Optimizing peer selection in BitTorrent networks with genetic algorithms. Future Generation Computer Systems. 26 (8), pp. 1151-1156. https://doi.org/10.1016/j.future.2010.05.016
Automatically wrapping legacy software into services: a grid case study
Li, M., Yu, B., Qi, M. and Antonopoulos, N. 2008. Automatically wrapping legacy software into services: a grid case study. Peer-to-Peer Networking and Applications. 1 (2), pp. 139-147. https://doi.org/10.1007/s12083-008-0011-9
Tracking online trails
Qi, M., Edgar-Nevill, D., Wang, Y. and Xu, R. 2008. Tracking online trails. International Journal of Electronic Security and Digital Forensics. 1 (4), pp. 353-361. https://doi.org/10.1504/IJESDF.2008.021453
Social networking searching and privacy issues
Qi, M. and Edgar-Nevill, D. 2011. Social networking searching and privacy issues. Information Security Technical Report. 16 (2), pp. 74-78. https://doi.org/10.1016/j.istr.2011.09.005
A WSRF based shopping cart system
Li, M., Qi, M., Rozati, M. and Yu, B. 2005. A WSRF based shopping cart system. Lecture Notes in Computer Science [Advances in Grid Computing (EGC) 2005 European Grid Conference, Amsterdam, The Netherlands, February 14-16, 2005, Revised Selected Papers]. 3470, pp. 105-137. https://doi.org/10.1007/b137919
Fighting cybercrime: legislation in China
Qi, M., Wang, Y. and Xu, R. 2009. Fighting cybercrime: legislation in China. International Journal of Electronic Security and Digital Forensics. 2 (2), pp. 219-227. https://doi.org/10.1504/IJESDF.2009.024905
Service composition with Al planning
Qi, M. 2009. Service composition with Al planning. in: Li, M. and Yu, B. (ed.) Cyberinfrastructure Technologies and Applications Nova Science Publishers. pp. 179-197