Scalable Distributed Computing and Intelligent Signal Processing for Massive IoT Data Streams
Scalable Distributed Computing and Intelligent Signal Processing for Massive IoT Data Streams |
||
 |
 |
|
| © 2024 by IJETT Journal | ||
| Volume-72 Issue-11 |
||
| Year of Publication : 2024 | ||
| Author : Balaji C G, Madhavi Damle, Abhijit Chirputkar |
||
| DOI : 10.14445/22315381/IJETT-V72I11P125 | ||
How to Cite?
Balaji C G, Madhavi Damle, Abhijit Chirputkar, "Scalable Distributed Computing and Intelligent Signal Processing for Massive IoT Data Streams," International Journal of Engineering Trends and Technology, vol. 72, no. 11, pp. 244-256, 2024. Crossref, https://doi.org/10.14445/22315381/IJETT-V72I11P125
Abstract
The Internet of Things (IoT) has complemented an era of unprecedented data generation, with billions of connected devices producing massive streams of sensor-generated data. This paper presents a comprehensive framework for IoT-driven signal processing, addressing the challenges of extracting meaningful patterns and insights from these vast and heterogeneous data streams. We propose a multi-layered approach that integrates advanced signal processing techniques with distributed computing paradigms and machine learning algorithms. Our framework encompasses adaptive sampling and compression methods to optimize data acquisition, distributed processing algorithms for scalable analysis, and novel machine learning techniques tailored to the dynamic nature of IoT data. We introduce a lightweight convolutional neural network architecture for edge computing, an online learning algorithm with concept drift adaptation, and a tensor-based fusion method for multi-modal data integration. Extensive experimental results demonstrate the efficacy of our proposed framework across various IoT scenarios, including smart cities, industrial IoT, and healthcare monitoring systems. Our adaptive sampling technique achieved up to 62.8% data reduction while maintaining 97.5% information preservation. The distributed processing approaches showed excellent scalability, with near-linear speedup for up to 64 nodes. The machine learning methodologies demonstrated superior performance in pattern recognition and anomaly detection tasks, with our lightweight CNN achieving 93.8% accuracy while reducing parameters by 75% compared to standard architectures.
Keywords
Internet of Things (IoT), Signal processing, Machine learning, Distributed computing, Data security and Privacy.
References
[1] Muhammad Ali Jamshed et al., “Challenges, Applications, and Future of Wireless Sensors in the Internet of Things: A Review,” IEEE Sensors Journal, vol. 22, no. 6, pp. 5482-5494, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[2] Rajalakshmi Krishnamurthi et al., “An Overview of IoT Sensor Data Processing, Fusion, and Analysis Techniques,” Sensors, vol. 20, no. 21, pp. 1-23, 2020.
[CrossRef] [Google Scholar] [Publisher Link]
[3] Anshu Shukla, and Yogesh Simmhan, “Benchmarking Distributed Stream Processing Platforms for IoT Applications,” Performance Evaluation and Benchmarking. Traditional - Big Data - Internet of Things: 8th TPC Technology Conference, New Delhi, India, pp 90-106, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[4] Adeyinka Akanbi, “ESTemd: A Distributed Processing Framework for Environmental Monitoring based on Apache Kafka Streaming Engine,” Proceedings of the 4th International Conference on Big Data Research, Tokyo Japan, pp. 18-25, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[5] Kun Lan et al., “Self-Adaptive Pre-Processing Methodology for Big Data Stream Mining in Internet of Things Environmental Sensor Monitoring,” Symmetry, vol. 9, no. 10, pp. 1-17, 2017.
[CrossRef] [Google Scholar] [Publisher Link]
[6] Geetanjali Rathee et al., “TrustSys: Trusted Decision Making Scheme for Collaborative Artificial Intelligence of Things,” IEEE Transactions on Industrial Informatics, vol. 19, no. 1, pp. 1059-1068, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[7] Ziyu Wan et al., “KFIML: Kubernetes- Based Fog Computing IoT Platform for Online Machine Learning,” IEEE Internet of Things Journal, vol. 9, no. 19, pp. 19463-19476, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[8] Zhaohong Wang, and Jing Guo, “Denoising Signals on the Graph for Distributed Systems by Secure Outsourced Computation,” 2021 IEEE 7th World Forum on Internet of Things (WF-IoT), New Orleans, LA, USA, pp. 524-529, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[9] Jinlai Xu, Balaji Palanisamy, and Qingyang Wang, “Resilient Stream Processing in Edge Computing,” 2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid), Melbourne, Australia, pp. 504-513, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[10] Yilin Yang et al., “Secure Coded Computation for Efficient Distributed Learning in Mobile IoT,” 2021 18th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), Rome, Italy, pp. 1-9, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[11] Aluizio Rocha Neto et al., “Optimizing Resource Allocation in Edge-distributed Stream Processing,” Proceedings of the 17th International Conference on Web Information Systems and Technologies WEBIST, vol. 1, pp. 156-166, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[12] Zhihan Lv et al., “AI-Enabled IoT-Edge Data Analytics for Connected Living,” ACM Transactions on Internet Technology, vol. 21, no. 4, pp. 1-20, 2021.
[CrossRef] [Google Scholar] [Publisher Link]
[13] Nwe Ni Hlaing, and Thi Thi Soe Nyunt, “Developing Scalable and Lightweight Data Stream Ingestion Framework for Stream Processing,” 2023 IEEE Conference on Computer Applications (ICCA), Yangon, Myanmar, pp. 405-410, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[14] Emna Baccour et al., “Pervasive AI for IoT Applications: A Survey on Resource-Efficient Distributed Artificial Intelligence,” IEEE Communications Surveys & Tutorials, vol. 24, no. 4, pp. 2366-2418, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[15] Wanli Ni, Jingheng Zheng, and Hui Tian, “Semi-Federated Learning for Collaborative Intelligence in Massive IoT Networks,” IEEE Internet of Things Journal, vol. 10, no. 13, pp. 11942-11943, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[16] Gul Agha, Dipayan Mukherjee, and Atul Sandur, “Performance, Energy and Parallelism: Using Near Data Processing in Utility and Cloud Computing,” 2022 IEEE/ACM 15th International Conference on Utility and Cloud Computing, Vancouver, WA, USA, pp. 173-180, 2022.
[CrossRef] [Google Scholar] [Publisher Link]
[17] Xu Liu et al., “LargeST: A Benchmark Dataset for Large-Scale Traffic Forecasting,” Proceedings of the 37th International Conference on Neural Information Processing Systems, New Orleans LA USA, pp. 75354-75371, 2024.
[Google Scholar] [Publisher Link]
[18] Narjes Davari et al., “MetroPT-3 Dataset,” UCI Machine Learning Repository, 2023.
[CrossRef] [Google Scholar] [Publisher Link]
[19] Nupur Biswas, and Shashaanka Ashili, “Smartwatch Heart Rate Data,” IEEE Dataport, 2023.
[CrossRef] [Publisher Link]