Denial of Service (DOS) Attack Detection on MQTT Protocol Using the Random Forest Method

Kharisma Monika Dian Pertiwi; Nurul Azizi Hasibuan; Dyah Putri Rahmawati

doi:10.15575/join.v11i1.1784

Authors

Kharisma Monika Dian Pertiwi Department of Informatics, Surabaya Campus Directorate, Telkom University, Indonesia
Nurul Azizi Hasibuan Department of Informatics, Surabaya Campus Directorate, Telkom University, Indonesia
Dyah Putri Rahmawati Department of Informatics, Surabaya Campus Directorate, Telkom University, Indonesia

DOI:

https://doi.org/10.15575/join.v11i1.1784

Keywords:

Attack detection, Denial of servive (DoS), IoT security, MQTT, Random forest

Abstract

The Message Queuing Telemetry Transport (MQTT) protocol serves as a critical lightweight communication infrastructure for Internet of Things (IoT) systems. Still, it remains highly vulnerable to Denial of Service (DoS) attacks that compromise network availability and security. Despite extensive IoT security research, existing MQTT-based intrusion detection systems predominantly employ binary classification approaches and lack comprehensive multi-class attack differentiation capabilities, limiting their practical deployment in real-world scenarios. This study addresses this critical gap by developing a multi-class DoS attack detection system utilizing the Random Forest algorithm to simultaneously classify normal traffic, MQTT flooding attacks, and SYN flood attacks. The methodology encompasses four systematic stages: collecting an MQTT network traffic dataset containing 1,634,286 records across three attack categories through controlled simulations; performing rigorous data preprocessing for cleaning and normalization; strategically extracting 60 MQTT-specific attributes to identify attack signatures; and implementing Random Forest with optimized hyperparameters for multi-class classification. Experimental results demonstrate optimal performance using an 80:20 train-test split with 5-fold cross-validation, achieving 95.27% precision, 95.09% recall, 95.08% F1-score, and 95.09% accuracy. A comprehensive evaluation using macro and micro-averaged metrics confirms the model's ability to autonomously classify MQTT network traffic types with high accuracy and balanced performance across all attack categories, offering a practical security solution for MQTT-enabled IoT infrastructure.

References

[1] X. Xie, Z. Jin, J. Wang, L. Yang, Y. Lu, and T. Li, “Confidence guided anomaly detection model for anti-concept drift in dynamic logs,” Journal of Network and Computer Applications, vol. 162, p. 102659, Jul. 2020, doi: 10.1016/j.jnca.2020.102659.

[2] S. Megawati and A. Lawi, “Pengembangan Sistem Teknologi Internet of Things Yang Perlu Dikembangkan Negara Indonesia,” 2021.

[3] C. B. V. Ilea and E. M. Carlini, Optimal Computation of Network Indicators for Electricity Market Bidding Zones Configuration. IEEE, 2020.

[4] N. Kamiya, “Corrections to ‘Learning-Based Signal Detection for Wireless OAM-MIMO Systems With Uniform Circular Array Antennas,’” IEEE Access, vol. 9, pp. 19818–19818, 2021, doi: 10.1109/ACCESS.2021.3054567.

[5] E. Bertino, “Data Security and Privacy in the IoT.,” EDBT, 2016, [Online]. Available: https://www.cs.purdue.edu/homes/bertino/paper-a.pdf0.pdf

[6] J. Su, S. He, and Y. Wu, “Features selection and prediction for IoT attacks,” High-Confidence Computing, vol. 2, no. 2, Jun. 2022, doi: 10.1016/j.hcc.2021.100047.

[7] F. Antony and R. Gustriansyah, “Deteksi Serangan Denial of Service pada Internet of Things Menggunakan Finite-State Automata,” MATRIK : Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer, vol. 21, no. 1, pp. 43–52, Nov. 2021, doi: 10.30812/matrik.v21i1.1078.

[8] G. M. I. Akbar, M. A. Hariyadi, and A. Hanani, “Detection of Bruteforce Attacks on the MQTT Protocol Using Random Forest Algorithm,” Internet of Things and Artificial Intelligence Journal, vol. 3, no. 3, pp. 250–272, Aug. 2023, doi: 10.31763/iota.v3i3.630.

[9] N. F. Syed, Z. Baig, A. Ibrahim, and C. Valli, “Denial of service attack detection through machine learning for the IoT,” Journal of Information and Telecommunication, vol. 4, no. 4, pp. 482–503, 2020, doi: 10.1080/24751839.2020.1767484.

[10] N. D. Primadya, A. Nugraha, A. Luthfiarta, and S. Y. Fahrezi, “Optimasi Logistic Regression untuk Deteksi Serangan DoS pada Keamanan IoT,” Jurnal Eksplora Informatika, vol. 13, no. 2, pp. 245–252, Mar. 2024, doi: 10.30864/eksplora.v13i2.1065.

[11] A. Alabdulatif, N. N. Thilakarathne, and M. Aashiq, “Machine Learning Enabled Novel Real-Time IoT Targeted DoS/DDoS Cyber Attack Detection System,” Computers, Materials and Continua, vol. 80, no. 3, pp. 3655–3683, 2024, doi: 10.32604/cmc.2024.054610.

[12] F. F. Setiadi, M. W. A. Kesiman, and K. Y. E. Aryanto, “Detection of dos attacks using naive bayes method based on internet of things (iot),” in Journal of Physics: Conference Series, IOP Publishing Ltd, Mar. 2021. doi: 10.1088/1742-6596/1810/1/012013.

[13] K. J. Harnanta, A. Bhawiyuga, and A. Basuki, “Implementasi MQTT Broker dengan Kemampuan Auto Scaling pada Internet of Things,” 2020. [Online]. Available: http://j-ptiik.ub.ac.id

[14] R. A. Khairulah, R. Herdianto, and M. A. Setiawan, “Klasifikasi Serangan Pada Jaringan Internet of Thing (IoT): Tinjauan Literatur Komparatif,” Jurnal Inovasi Teknik dan Edukasi Teknologi, vol. 3, no. 1, pp. 47–53, doi: 10.17977/um068v3i12023p47-53.

[15] F. T. Kurdi, “Random Forest Machine Learning Technique for Automatic Vegetation Detection and Modelling in LiDAR Data,” International Journal of Environmental Sciences & Natural Resources, vol. 28, no. 2, Jun. 2021, doi: 10.19080/ijesnr.2021.28.556234.

[16] A. Zilham Fadillah and R. Gunawan, “POTENSI IOT DALAM INDUSTRI 4.0,” 2024.

[17] I. Latin and A. Transactions, “MQTT Protocol: Fundamentals, Tools and Future Directions,” 2019.

[18] R. A. Radouan Ait Mouha, “Internet of Things (IoT),” Journal of Data Analysis and Information Processing, vol. 09, no. 02, pp. 77–101, 2021, doi: 10.4236/jdaip.2021.92006.

[19] A. Mcewen and H. Cassimally, “Designing the Internet of Things.”

[20] E. Longo, A. E. C. Redondi, and M. Casana, MQTT-ST: a Spanning Tree Protocol for Distributed MQTT Brokers. IEEE, 2020.

[21] Al Hanif and R. Amri, “Implementasi Internet Of Things Pada Protokol MQTT Dan HTTP Dalam Sistem Pendeteksi Banjir,” JURNAL INOVTEK POLBENG, vol. 8, no. 2, 2023.

[22] R. Poorzare and A. C. Augé, “Challenges on the way of implementing TCP over 5G networks,” IEEE Access, vol. 8, pp. 176393–176415, 2020, doi: 10.1109/ACCESS.2020.3026540.

[23] R. Tandon, “A Survey of Distributed Denial of Service Attacks and Defenses,” Aug. 2020, [Online]. Available: http://arxiv.org/abs/2008.01345

[24] I. Wijayana and M. Ardiansyah, “Optimizing Qur’an Recitation Monitoring with Random Forest Algorithm,” Khazanah Journal of Religion and Technology, vol. 2, no. 1, pp. 1–5, Jun. 2024, doi: 10.15575/kjrt.v2i1.227.

[25] P. Kumala Sari and R. Randy Suryono, “KOMPARASI ALGORITMA SUPPORT VECTOR MACHINE DAN RANDOM FOREST UNTUK ANALISIS SENTIMEN METAVERSE,” 2024.

[26] R. Tri Novita, I. Gunawan, I. Marleni, O. Gregarius Grasia, and M. Nanda Valentika abcde Teknik Elektro Sekolah Tinggi Teknologi Ronggolawe Cepu Penulis Korenspondensi, “Analisis Keamanan Wifi Menggunakan Wireshark,” 2021. [Online]. Available: https://www.researchgate.net/publication/316464159_Analisis_Keama

[27] J. Lorincz, Z. Klarin, and J. Ožegović, “A comprehensive overview of tcp congestion control in 5g networks: Research challenges and future perspectives,” Jul. 01, 2021, MDPI AG. doi: 10.3390/s21134510.

[28] A. Primajaya and B. N. Sari, “Random Forest Algorithm for Prediction of Precipitation,” Indonesian Journal of Artificial Intelligence and Data Mining (IJAIDM), vol. 1, no. 1, pp. 27–31, 2018.

[29] A. Primajaya and B. N. Sari, “Random Forest Algorithm for Prediction of Precipitation,” Indonesian Journal of Artificial Intelligence and Data Mining (IJAIDM), vol. 1, no. 1, pp. 27–31, 2018.

[30] L. Qadrini, “Undersampling dan K-Fold Random Forest Untuk Klasifikasi Kelas Tidak Seimbang,” Building of Informatics, Technology and Science (BITS), vol. 4, no. 4, Mar. 2023, doi: 10.47065/bits.v4i4.3141.

[31] M. Arief, P. Hari Trisnawan, and M. Data, “IMPLEMENTASI SISTEM DETEKSI SERANGAN SLOWLORIS PADA ARSITEKTUR JARINGAN SOFTWARE-DEFINED NETWORK MENGGUNAKAN RANDOM FOREST,” 2017. [Online]. Available: http://j-ptiik.ub.ac.id

[32] S. A. Laghari, S. Manickam, and S. Karuppayah, “A Review on SECS/GEM: A Machine-to-Machine (M2M) Communication Protocol for Industry 4.0,” International Journal of Electrical and Electronic Engineering and Telecommunications, vol. 10, no. 2, pp. 105–114, Mar. 2021, doi: 10.18178/ijeetc.10.2.105-114.

[33] D. Silva, L. I. Carvalho, J. Soares, and R. C. Sofia, “A performance analysis of internet of things networking protocols: Evaluating MQTT, CoAP, OPC UA,” Applied Sciences (Switzerland), vol. 11, no. 11, Jun. 2021, doi: 10.3390/app11114879.

[34] H. Zi Rui, T. Ying Chien, L. Xin Ee, C. Wei San, and L. Teng Yi, “Comparison of the Use of Support Vector Machine (SVM) & Random Forest Algorithms (RF) for DDOS Attack Detection”, doi: 10.47772/IJRISS.

[35] Y. Irawan, R. Pramitasari, W. M. Ashari, A. Nur, and H. Yansyah, “Support Vector Machine Classification Algorithm for Detecting DDoS Attacks on Network Traffic,” 2025. [Online]. Available: http://jurnal.polibatam.ac.id/index.php/JAIC

[36] A. Hakeem and A. Attiah, “Machine Learning-based Approach for Detecting DDoS Attacks in Software Defined Networks,” 2024.