A resource aware MapReduce based parallel SVM for large scale image classification

Journal article

Guo, W., Khalid, N., Liu, Y., Li, M., Qi, M., Guo, W., Khalid, N., Liu, Y., Li, M. and Qi, M. 2015. A resource aware MapReduce based parallel SVM for large scale image classification. Neural Processing Letters. 44 (1), pp. 161-184.
AuthorsGuo, W., Khalid, N., Liu, Y., Li, M., Qi, M., Guo, W., Khalid, N., Liu, Y., Li, M. and Qi, M.
Abstract

Machine learning techniques have facilitated image retrieval by automatically classifying and annotating images with keywords. Among them support vector machines (SVMs) are used extensively due to their generalization properties. However, SVM training is notably a computationally intensive process especially when the training dataset is large.

This paper presents RASMO, a resource aware MapReduce based parallel SVM algorithm for large scale image classifications which partitions the training data set into smaller subsets and optimizes SVM training in parallel using a cluster of computers. A genetic algorithm based load balancing scheme is designed to optimize the performance of RASMO in heterogeneous computing environments. RASMO is evaluated in both experimental and simulation environments.

The results show that the parallel SVM algorithm reduces the training time significantly compared with the sequential SMO algorithm while maintaining a high level of accuracy in classifications

KeywordsParallel SVM; MapReduce; image classification and annotation; load balancing
Year2015
JournalNeural Processing Letters
Journal citation44 (1), pp. 161-184
PublisherKluwer
ISSN1370-4621
Official URLhttps://dl.acm.org/citation.cfm?id=J580&picked=prox
Publication dates
Online18 Sep 2015
Publication process dates
Deposited03 Apr 2018
Accepted author manuscript
Output statusPublished
Permalink -

https://repository.canterbury.ac.uk/item/88q11/a-resource-aware-mapreduce-based-parallel-svm-for-large-scale-image-classification

Download files

Accepted author manuscript
17177_A Resource Aware MapReduce based Parallel SVM for Large Scale Image Classifications (2).pdf

  • 60
    total views
  • 12
    total downloads
  • 0
    views this month
  • 0
    downloads this month

Export as

Related outputs

Pre processing of social media remarks for forensics
Qi, M. and Gao, X. 2023. Pre processing of social media remarks for forensics.
Alignment-based conformance checking if hierarchical process models
Qi, M., Wang, L, Han, X., Wang, K. and Lu, P. 2023. Alignment-based conformance checking if hierarchical process models. Computing and Informatics.
Research on UBI auto insurance pricing model based on parameter adaptive SAPSO optimal fuzzy controller
Wang, X., Liu, W., Ou, Z., Yan, C., Qi, M. and Jia, W. 2022. Research on UBI auto insurance pricing model based on parameter adaptive SAPSO optimal fuzzy controller. Computing and Informatics. 41 (4), pp. 1078-1113. https://doi.org/10.31577/cai_2022_4_1078
Novel architecture for human re-identification with a two-stream neural network and attention ,echanism
Rahi, B. and Qi, M. 2022. Novel architecture for human re-identification with a two-stream neural network and attention ,echanism. Computing and Informatics. 41 (4), pp. 905-930. https://doi.org/10.31577/cai_2022_4_905
A novel data analytic model for mining user insurance demands from microblogs
Chun Yan, Lu Liu, Wei Liu and Man Qi 2022. A novel data analytic model for mining user insurance demands from microblogs. Computing and Informatics. 41 (3). https://doi.org/10.31577/cai_2022_3_689
Deep learning based real-time facial mask detection and crowd monitoring
Chan-Yun Yang, Hooman Samani, Nana Ji, Chunxu Li, Ding-Bang Chen and Man Qi 2022. Deep learning based real-time facial mask detection and crowd monitoring. Computing and Informatics. 40 (6), pp. 1263-11294. https://doi.org/10.31577/cai_2021_6_1263
Repairing process models with non-free-choice constructs based on token replay
Bai, E., Qi, M., Luan, W., Li, P. and Du, Y. 2022. Repairing process models with non-free-choice constructs based on token replay. Computing and Informatics. 41 (4), pp. 1054-1077. https://doi.org/10.31577/cai_2022_4_1054
Transformers only look once with nonlinear combination for real-time object detection
Xia, R., Li, G., Huang, Z., Pang, Y. and Qi, M. 2022. Transformers only look once with nonlinear combination for real-time object detection. Neural Computing and Applications. https://doi.org/10.1007/s00521-022-07333-y
Deviation detection in clinical pathways based on business alignment
Tian, Y., Li, X., Qi, Man, Han, D. and Du, Yuyue 2022. Deviation detection in clinical pathways based on business alignment. Scientific Programming. 2022, pp. 1-13. https://doi.org/10.1155/2022/6993449
Security vulnerabilities of popular smart home appliances
Qi, M., Induruwa, A. and Hussain, F. 2021. Security vulnerabilities of popular smart home appliances. in: Proceeding of The Twentieth International Conference on Networks April 18, 2021 to April 22, 2021 - Porto, Portugal
Improved adaptive genetic algorithm for the vehicle insurance fraud identification model based on a BP neural network
Yan, C., Li, M., Liu, W. and Qi, M. 2020. Improved adaptive genetic algorithm for the vehicle insurance fraud identification model based on a BP neural network. Theoretical Computer Science. 817, pp. 12-23. https://doi.org/10.1016/j.tcs.2019.06.025
Hybrid Intrusion Detection System for Smart Home Applications
Hussain, Fida, Induruwa, Abhaya and Qi, Man 2020. Hybrid Intrusion Detection System for Smart Home Applications. in: Mahmood, Z. (ed.) Developing and Monitoring Smart Environments for Intelligent Cities IGI Global. pp. 300-322
Payments per claim model of outstanding claims reserve based on fuzzy linear regression
Yan, C., Liu, Q., Liu, J., Liu, W., Li, M. and Qi, M. 2019. Payments per claim model of outstanding claims reserve based on fuzzy linear regression. International Journal of Fuzzy Systems. 21, pp. 1950-1960. https://doi.org/10.1007/s40815-019-00617-x
Fuzzy interacting multiple model H∞ particle filter algorithm based on current statistical model
Wang, Q., Chen, X., Zhang, L., Li, J., Zhao, C. and Qi, M. 2019. Fuzzy interacting multiple model H∞ particle filter algorithm based on current statistical model. International Journal of Fuzzy Systems. 21, pp. 1894-1905. https://doi.org/10.1007/s40815-019-00678-y
Temporal sparse feature auto-combination deep network for video action recognition
Wang, Q., Gong, D., Qi, M., Shen, Y. and Lei, Y. 2018. Temporal sparse feature auto-combination deep network for video action recognition. Concurrency and Computation: Practice and Experience. https://doi.org/10.1002/cpe.4487
Soundness analytics of composed logical workflow nets
Liu, W., Wang, L., Feng, X., Qi, M., Yan, C. and Li, M. 2017. Soundness analytics of composed logical workflow nets. International Journal of Parallel Programming. https://doi.org/10.1007/s10766-017-0536-8
A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters
Liu, Y., Jing, W., Liu, Y., Lv, L., Qi, M. and Xiang, Y. 2016. A sliding window-based dynamic load balancing for heterogeneous Hadoop clusters. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3763
Gaussian-Gamma collaborative filtering: a hierarchical Bayesian model for recommender systems
Luo, C., Zhang, B., Xiang, Y. and Qi, M. 2017. Gaussian-Gamma collaborative filtering: a hierarchical Bayesian model for recommender systems. Journal of Computer and System Sciences. https://doi.org/10.1016/j.jcss.2017.03.007
Facilitating visual surveillance with motion detections
Qi, M. 2017. Facilitating visual surveillance with motion detections. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3770
Data security of android applications
Obiri-Yeboah, J. and Qi, M. 2016. Data security of android applications. in: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery : ICNC-FSKD 2016 : 13-15 August, Changsha, China IEEE Xplore.
AL-DDCNN : a distributed crossing semantic gap learning for person re-identification
Cheng, K., Zhan, Y. and Qi, M. 2017. AL-DDCNN : a distributed crossing semantic gap learning for person re-identification. Concurrency and Computation: Practice and Experience. 29 (3). https://doi.org/10.1002/cpe.3766
Integration of link and semantic relations for information recommendation
Zhao, Q., He, Y., Wang, P., Jiang, C. and Qi, M. 2016. Integration of link and semantic relations for information recommendation. Computing and Informatics. 35 (1), pp. 30-54.
Data-driven pedestrian re-identification based on hierarchical semantic representation
Cheng, K., Xu, F., Tao, F., Qi, M. and Li, M. 2018. Data-driven pedestrian re-identification based on hierarchical semantic representation. Concurrency and Computation: Practice and Experience. 30 (23). https://doi.org/10.1002/cpe.4403
Integrated privacy preserving framework for smart home
Qi, M. and Hussain, F. 2018. Integrated privacy preserving framework for smart home. in: Proceedings of The 14th International Conference on Fuzzy Systems and Knowledge Discovery, 28-30 July 2018 in Huangshan, China.
Parallelizing abnormal event detection in crowded scenes with GPU
Yavari, M., Li, M., Li, S. and Qi, M. 2015. Parallelizing abnormal event detection in crowded scenes with GPU.
Big data management in digital forensics
Qi, M. 2014. Big data management in digital forensics. in: 2014 IEEE 17th International Conference on Computational Science and Engineering (CSE) IEEE. pp. 238-243
Digital forensics and NoSQL databases
Qi, M. 2014. Digital forensics and NoSQL databases. in: Proceedings of the 11th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), 2014 IEEE. pp. 734-739
Employing neural networks for DDoS detection
Qi, M. 2014. Employing neural networks for DDoS detection.
A MapReduce based distributed LSI for scalable information retrieval
Liu, Y., Li, M., Khan, M. and Qi, M. 2014. A MapReduce based distributed LSI for scalable information retrieval. Computing and Informatics. 33 (2), pp. 259-280.
Clear up grey areas: cybercrime legislation progress in China
Qi, M. 2013. Clear up grey areas: cybercrime legislation progress in China.
Social media in law enforcement: the role and issues
Qi, M. 2012. Social media in law enforcement: the role and issues.
A distributed storage system for archiving broadcast media content
Cherry, D., Li, M. and Qi, M. 2012. A distributed storage system for archiving broadcast media content. in: Grid and Cloud Computing: Concepts, Methodologies, Tools and Applications IGI Global. pp. 669-679
A distributed SVM ensemble for image classification and annotation
Alham, M., Li, M., Liu, Y., Ponraj, M. and Qi, M. 2012. A distributed SVM ensemble for image classification and annotation. in: 2012 9th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) IEEE. pp. 1581-1584
Computer forensics in communication networks
Wang, Y. and Qi, M. 2012. Computer forensics in communication networks. in: IET International Communication Conference on Wireless Mobile and Computing (CCWMC 2011) Curran.
A MapReduce based parallel SVM for large scale spam filtering
Caruana, G., Li, M. and Qi, M. 2011. A MapReduce based parallel SVM for large scale spam filtering. in: Caruana, G. (ed.) Eighth International Conference on Fuzzy Systems and Knowledge Discovery (FSKD), 2011 IEEE.
Cyber safety in schools
Qi, M. 2011. Cyber safety in schools.
The social effects of spam
Mousoli, R., Qi, M. and Edgar-Nevill, D. 2008. The social effects of spam. in: Edgar-Nevill, D. (ed.) Proceedings of CFET 2008 – 2nd Annual International Conference on Cybercrime Forensics Education and Training Canterbury Canterbury Christ Church University. pp. 71-78
A distributed SVM ensemble for large scale image classification and annotation
Alham, K., Li, M., Liu, Y. and Qi, M. 2013. A distributed SVM ensemble for large scale image classification and annotation. Computers and Mathematics with Applications. 66 (10), pp. 1920-1934. https://doi.org/10.1016/j.camwa.2013.07.015
Fast implementation of cel-based animation
Qi, M. 2008. Fast implementation of cel-based animation.
A case-based reasoning model for digital intrusion forensics
Qi, M., Wang, Y. and Xu, R. 2008. A case-based reasoning model for digital intrusion forensics.
Cybercrime legislation in China
Qi, M., Wang, Y. and Xu, R. 2008. Cybercrime legislation in China.
P2P DDoS: Challenges and Countermeasures
Qi, M. 2009. P2P DDoS: Challenges and Countermeasures.
Distributed indexing for resource discovery in P2P(Peer to Peer) networks
Qi, M., Hentschel, M., Li, M. and Ponraj, M. 2009. Distributed indexing for resource discovery in P2P(Peer to Peer) networks.
P2P(peer to peer) network-targeted DDoS(distributed denial-of-service attacks)
Qi, M. 2009. P2P(peer to peer) network-targeted DDoS(distributed denial-of-service attacks).
Spam and social effects
Qi, M., Edgar-Nevill, D. and Mousoli, R. 2009. Spam and social effects.
Footprints of cyber criminals
Qi, M. and Edgar-Nevill, D. 2007. Footprints of cyber criminals.
Tracking email offenders
Edgar-Nevill, D. and Qi, M. 2007. Tracking email offenders.
Protecting intellectual property and computer software: legislation in China
Qi, M. and Wang, Y. 2010. Protecting intellectual property and computer software: legislation in China.
Machine learning based spam filtering: advantages and challenges
Qi, M. 2010. Machine learning based spam filtering: advantages and challenges.
OBIRE: ontology based bibliographic information publication and retrieval
Liu, X., Li, M., Liu, Y. and Qi, M. 2010. OBIRE: ontology based bibliographic information publication and retrieval. International Journal of Distributed Systems and Technologies. 1 (4), pp. 58-73.
Cybercrime and relevant legislation in China
Qi, M. and Wang, Y. 2010. Cybercrime and relevant legislation in China. in: Reich, P. (ed.) Cybercrime and Security Oxford Oxford University Press. pp. 1-34
Semantic analysis for spam filtering
Qi, M. and Mousoli, R. 2010. Semantic analysis for spam filtering.
MAPBOT: A web based map information retrieval system
Li, M. and Qi, M. 2003. MAPBOT: A web based map information retrieval system. Information and Software Technology. 45 (10), pp. 691-698. https://doi.org/10.1016/S0950-5849(03)00082-X
Leveraging legacy codes to distributed problem-solving environments: a web services approach
Li, M. and Qi, M. 2004. Leveraging legacy codes to distributed problem-solving environments: a web services approach. Software: Practice and Experience. 34 (13), pp. 1297-1309. https://doi.org/10.1002/spe.614
PGGA: a predictable and grouped genetic algorithm for job scheduling
Li, M., Yu, B. and Qi, M. 2006. PGGA: a predictable and grouped genetic algorithm for job scheduling. Future Generation Computer Systems. 22 (5), pp. 588-599. https://doi.org/10.1016/j.future.2005.09.001
Lessons learned from Beijing for the London 2012
Edgar-Nevill, D. and Qi, M. 2009. Lessons learned from Beijing for the London 2012. in: Edgar-Nevill, D. (ed.) Proceedings of the 3rd International Conference on Cybercrime Forensics Euducation and Training CFET 2009 Canterbury, UK Canterbury Christ Church University.
Tracking online trails
Qi, M., Edgar-Nevill, D., Wang, Y. and Xu, R. 2008. Tracking online trails. in: Jahankhani, H., Revett, K. and Palmer-Brown, D. (ed.) Global E-Security Springer.
Web services discovery with rough sets
Li, M., Yu, B., Sahota, V. and Qi, M. 2009. Web services discovery with rough sets. International Journal of Web Services Research. 6 (1), pp. 69-86.
Facilitating resource discovery in grid environments with peer-to-peer structured tuple spaces
Li, M. and Qi, M. 2009. Facilitating resource discovery in grid environments with peer-to-peer structured tuple spaces. Peer-to-Peer Networking and Applications. 2 (4), pp. 283-297. https://doi.org/10.1007/s12083-009-0036-8
Optimizing peer selection in BitTorrent networks with genetic algorithms
Wu, T., Li, M. and Qi, M. 2010. Optimizing peer selection in BitTorrent networks with genetic algorithms. Future Generation Computer Systems. 26 (8), pp. 1151-1156. https://doi.org/10.1016/j.future.2010.05.016
Automatically wrapping legacy software into services: a grid case study
Li, M., Yu, B., Qi, M. and Antonopoulos, N. 2008. Automatically wrapping legacy software into services: a grid case study. Peer-to-Peer Networking and Applications. 1 (2), pp. 139-147. https://doi.org/10.1007/s12083-008-0011-9
Tracking online trails
Qi, M., Edgar-Nevill, D., Wang, Y. and Xu, R. 2008. Tracking online trails. International Journal of Electronic Security and Digital Forensics. 1 (4), pp. 353-361. https://doi.org/10.1504/IJESDF.2008.021453
Social networking searching and privacy issues
Qi, M. and Edgar-Nevill, D. 2011. Social networking searching and privacy issues. Information Security Technical Report. 16 (2), pp. 74-78. https://doi.org/10.1016/j.istr.2011.09.005
A WSRF based shopping cart system
Li, M., Qi, M., Rozati, M. and Yu, B. 2005. A WSRF based shopping cart system. Lecture Notes in Computer Science [Advances in Grid Computing (EGC) 2005 European Grid Conference, Amsterdam, The Netherlands, February 14-16, 2005, Revised Selected Papers]. 3470, pp. 105-137. https://doi.org/10.1007/b137919
Fighting cybercrime: legislation in China
Qi, M., Wang, Y. and Xu, R. 2009. Fighting cybercrime: legislation in China. International Journal of Electronic Security and Digital Forensics. 2 (2), pp. 219-227. https://doi.org/10.1504/IJESDF.2009.024905
Service composition with Al planning
Qi, M. 2009. Service composition with Al planning. in: Li, M. and Yu, B. (ed.) Cyberinfrastructure Technologies and Applications Nova Science Publishers. pp. 179-197