Advancements in Deep Learning: A Comprehensive Survey on Architectures, Optimization Techniques, and Applications
DOI:
https://doi.org/10.63876/ijtm.v3i2.141Keywords:
Deep Learning, Neural Network Architectures, Optimization Techniques, Machine Learning Applications, Artificial Intelligence, Transformers and CNNsAbstract
Deep learning has revolutionized the field of artificial intelligence by enabling significant advancements across various domains, including computer vision, natural language processing, and speech recognition. This survey provides a comprehensive overview of recent developments in deep learning, focusing on three core aspects: architectural innovations, optimization strategies, and real-world applications. We explore the evolution of neural network architectures, from classical feedforward networks to cutting-edge models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, and graph neural networks (GNNs). In addition, we examine state-of-the-art optimization techniques, including adaptive learning rate methods, regularization strategies, and training heuristics that address challenges like vanishing gradients and overfitting. Finally, we present a broad spectrum of deep learning applications, highlighting breakthroughs in autonomous systems, healthcare, finance, and more. By synthesizing recent research trends and identifying emerging challenges, this survey aims to serve as a valuable resource for researchers and practitioners seeking to navigate the rapidly evolving landscape of deep learning.
Downloads
References
L.-H. Wen and K.-H. Jo, “Deep learning-based perception systems for autonomous driving: A comprehensive survey,” Neurocomputing, vol. 489, pp. 255–270, Jun. 2022, doi: https://doi.org/10.1016/j.neucom.2021.08.155.
S. Kim, U. Lee, I. Lee, and N. Kang, “Idle vehicle relocation strategy through deep learning for shared autonomous electric vehicle system optimization,” J. Clean. Prod., vol. 333, p. 130055, Jan. 2022, doi: https://doi.org/10.1016/j.jclepro.2021.130055.
T. Okuyama, T. Gonsalves, and J. Upadhay, “Autonomous Driving System based on Deep Q Learnig,” in 2018 International Conference on Intelligent Autonomous Systems (ICoIAS), IEEE, Mar. 2018, pp. 201–205. doi: https://doi.org/10.1109/ICoIAS.2018.8494053.
H. Nai and H. Jiang, “Time Series Identification Method for Extracting Structural Features,” in 2019 IEEE 5th Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing, (HPSC) and IEEE Intl Conference on Intelligent Data and Security (IDS), IEEE, May 2019, pp. 321–327. doi: https://doi.org/10.1109/BigDataSecurity-HPSC-IDS.2019.00065.
G. Mendez and A. de Antonio, “A Modifiable Agent-Based Software Architecture for Intelligent Virtual Environments for Training,” in Seventh Working IEEE/IFIP Conference on Software Architecture (WICSA 2008), IEEE, Feb. 2008, pp. 319–322. doi: https://doi.org/10.1109/WICSA.2008.35.
A. Chatterjee, J. Saha, and J. Mukherjee, “Clustering with multi-layered perceptron,” Pattern Recognit. Lett., vol. 155, pp. 92–99, Mar. 2022, doi: https://doi.org/10.1016/j.patrec.2022.02.009.
M. Stankovic et al., “Tuning Multi-Layer Perceptron by Hybridized Arithmetic Optimization Algorithm for Healthcare 4.0,” Procedia Comput. Sci., vol. 215, pp. 51–60, 2022, doi: https://doi.org/10.1016/j.procs.2022.12.006.
H. Jin, Y.-G. Kim, Z. Jin, A. A. Rushchitc, and A. S. Al-Shati, “Optimization and analysis of bioenergy production using machine learning modeling: Multi-layer perceptron, Gaussian processes regression, K-nearest neighbors, and Artificial neural network models,” Energy Reports, vol. 8, pp. 13979–13996, Nov. 2022, doi: https://doi.org/10.1016/j.egyr.2022.10.334.
M. Stoffel, F. Bamer, and B. Markert, “Deep convolutional neural networks in structural dynamics under consideration of viscoplastic material behaviour,” Mech. Res. Commun., vol. 108, p. 103565, Sep. 2020, doi: https://doi.org/10.1016/j.mechrescom.2020.103565.
V. Vives-Boix and D. Ruiz-Fernández, “Synaptic metaplasticity for image processing enhancement in convolutional neural networks,” Neurocomputing, vol. 462, pp. 534–543, Oct. 2021, doi: https://doi.org/10.1016/j.neucom.2021.08.021.
G. Ortac and G. Ozcan, “Comparative study of hyperspectral image classification by multidimensional Convolutional Neural Network approaches to improve accuracy,” Expert Syst. Appl., vol. 182, p. 115280, Nov. 2021, doi: https://doi.org/10.1016/j.eswa.2021.115280.
C. Zhang, S. Li, M. Ye, C. Zhu, and X. Li, “Learning various length dependence by dual recurrent neural networks,” Neurocomputing, vol. 466, pp. 1–15, Nov. 2021, doi: https://doi.org/10.1016/j.neucom.2021.09.043.
D. Zhu and D. Wang, “Transformers and their application to medical image processing: A review,” J. Radiat. Res. Appl. Sci., vol. 16, no. 4, p. 100680, Dec. 2023, doi: https://doi.org/10.1016/j.jrras.2023.100680.
G. Renton, M. Balcilar, P. Héroux, B. Gaüzère, P. Honeine, and S. Adam, “Symbols Detection and Classification using Graph Neural Networks,” Pattern Recognit. Lett., vol. 152, pp. 391–397, Dec. 2021, doi: https://doi.org/10.1016/j.patrec.2021.09.020.
T. Wang and J. Knap, “Stochastic gradient descent for semilinear elliptic equations with uncertainties,” J. Comput. Phys., vol. 426, p. 109945, Feb. 2021, doi: https://doi.org/10.1016/j.jcp.2020.109945.
Y. Xue, Y. Tong, and F. Neri, “An ensemble of differential evolution and Adam for training feed-forward neural networks,” Inf. Sci. (Ny)., vol. 608, pp. 453–471, Aug. 2022, doi: https://doi.org/10.1016/j.ins.2022.06.036.
X. Hu, S. Wen, and H. K. Lam, “Dynamic random distribution learning rate for neural networks training,” Appl. Soft Comput., vol. 124, p. 109058, Jul. 2022, doi: https://doi.org/10.1016/j.asoc.2022.109058.
R. Pahič, B. Ridge, A. Gams, J. Morimoto, and A. Ude, “Training of deep neural networks for the generation of dynamic movement primitives,” Neural Networks, vol. 127, pp. 121–131, Jul. 2020, doi: https://doi.org/10.1016/j.neunet.2020.04.010.
C. Ha, V.-D. Tran, L. Ngo Van, and K. Than, “Eliminating overfitting of probabilistic topic models on short and noisy text: The role of dropout,” Int. J. Approx. Reason., vol. 112, pp. 85–104, Sep. 2019, doi: https://doi.org/10.1016/j.ijar.2019.05.010.
W. Pan, S. Tian, J. Li, W. Jin, L. Yuan, and Q. Yao, “Convolutional neural network-based multi-modal detection model for combustion instability in swirling flames,” Aerosp. Sci. Technol., vol. 162, p. 110243, Jul. 2025, doi: https://doi.org/10.1016/j.ast.2025.110243.
A. Mathis, S. Schneider, J. Lauer, and M. W. Mathis, “A Primer on Motion Capture with Deep Learning: Principles, Pitfalls, and Perspectives,” Neuron, vol. 108, no. 1, pp. 44–65, Oct. 2020, doi: https://doi.org/10.1016/j.neuron.2020.09.017.
C. Geng, L. Tao, and S. Chen, “Guided CNN for generalized zero-shot and open-set recognition using visual and semantic prototypes,” Pattern Recognit., vol. 102, p. 107263, Jun. 2020, doi: https://doi.org/10.1016/j.patcog.2020.107263.
A. Poznyak, I. Chairez, and T. Poznyak, “A survey on artificial neural networks application for identification and control in environmental engineering: Biological and chemical systems with uncertain models,” Annu. Rev. Control, vol. 48, pp. 250–272, 2019, doi: https://doi.org/10.1016/j.arcontrol.2019.07.003.
J. Á. González, L.-F. Hurtado, and F. Pla, “Transformer based contextualization of pre-trained word embeddings for irony detection in Twitter,” Inf. Process. Manag., vol. 57, no. 4, p. 102262, Jul. 2020, doi: https://doi.org/10.1016/j.ipm.2020.102262.
X. Wu, J. Lin, K. Zhang, and M. Cheng, “A switched dynamical system approach towards the optimal control of chemical processes based on a gradient-based parallel optimization algorithm,” Comput. Chem. Eng., vol. 118, pp. 180–194, Oct. 2018, doi: https://doi.org/10.1016/j.compchemeng.2018.08.007.
X. Zeng, L. Wen, Y. Xu, and C. Ji, “Generating diagnostic report for medical image by high-middle-level visual information incorporation on double deep learning models,” Comput. Methods Programs Biomed., vol. 197, p. 105700, Dec. 2020, doi: https://doi.org/10.1016/j.cmpb.2020.105700.
J. West and M. Bhattacharya, “Intelligent financial fraud detection: A comprehensive review,” Comput. Secur., vol. 57, pp. 47–66, Mar. 2016, doi: https://doi.org/10.1016/j.cose.2015.09.005.
V. Karunakar Reddy and R. Kumar AV, “Multi-channel neuro signal classification using Adam-based coyote optimization enabled deep belief network,” Biomed. Signal Process. Control, vol. 77, p. 103774, Aug. 2022, doi: https://doi.org/10.1016/j.bspc.2022.103774.
D. Xu, S. Zhang, H. Zhang, and D. P. Mandic, “Convergence of the RMSProp deep learning method with penalty for nonconvex optimization,” Neural Networks, vol. 139, pp. 17–23, Jul. 2021, doi: https://doi.org/10.1016/j.neunet.2021.02.011.
Z. Yan, J. Chen, R. Hu, T. Huang, Y. Chen, and S. Wen, “Training memristor-based multilayer neuromorphic networks with SGD, momentum and adaptive learning rates,” Neural Networks, vol. 128, pp. 142–149, Aug. 2020, doi: https://doi.org/10.1016/j.neunet.2020.04.025.
Y. Wang, J. Liu, X. Chang, J. Wang, and R. J. Rodríguez, “AB-FGSM: AdaBelief optimizer and FGSM-based approach to generate adversarial examples,” J. Inf. Secur. Appl., vol. 68, p. 103227, Aug. 2022, doi: https://doi.org/10.1016/j.jisa.2022.103227.
S. A., C. Baskar, and S. Shoba, “A HybridOpt approach for early Alzheimer’s Disease diagnostics with Ant Lion Optimizer (ALO),” Alexandria Eng. J., vol. 109, pp. 112–125, Dec. 2024, doi: https://doi.org/10.1016/j.aej.2024.08.089.
M. B. Tozak, P. G. Durgut, and M. T. Ayvaz, “Optimum identification of aquifer parameter zone structures using the SHuffled Ant Lion Optimization approach considering general form of the Voronoi tessellation,” J. Hydrol., vol. 640, p. 131683, Aug. 2024, doi: https://doi.org/10.1016/j.jhydrol.2024.131683.
F. Bérchez-Moreno, J. C. Fernández, C. Hervás-Martínez, and P. A. Gutiérrez, “Fusion of standard and ordinal dropout techniques to regularise deep models,” Inf. Fusion, vol. 106, p. 102299, Jun. 2024, doi: https://doi.org/10.1016/j.inffus.2024.102299.
L. Landrieu, H. Raguet, B. Vallet, C. Mallet, and M. Weinmann, “A structured regularization framework for spatially smoothing semantic labelings of 3D point clouds,” ISPRS J. Photogramm. Remote Sens., vol. 132, pp. 102–118, Oct. 2017, doi: https://doi.org/10.1016/j.isprsjprs.2017.08.010.
G. Xie et al., “DHOSGR: lncRNA-disease Association Prediction Based on Decay High-order Similarity and Graph-regularized Matrix Completion,” Curr. Bioinform., vol. 18, no. 1, pp. 92–104, Jan. 2023, doi: https://doi.org/10.2174/1574893618666221118092849.
E. Agliari, F. Alemanno, M. Aquaro, and A. Fachechi, “Regularization, early-stopping and dreaming: A Hopfield-like setup to address generalization and overfitting,” Neural Networks, vol. 177, p. 106389, Sep. 2024, doi: https://doi.org/10.1016/j.neunet.2024.106389.
D. Rivoir, I. Funke, and S. Speidel, “On the pitfalls of Batch Normalization for end-to-end video learning: A study on surgical workflow analysis,” Med. Image Anal., vol. 94, p. 103126, May 2024, doi: https://doi.org/10.1016/j.media.2024.103126.
H. Zhang, Y. Pu, Z. Zhao, Y. Li, X. Li, and R. Nie, “Towards diverse image-to-image translation via adaptive normalization layer and contrast learning,” Comput. Graph., vol. 123, p. 104017, Oct. 2024, doi: https://doi.org/10.1016/j.cag.2024.104017.
D. Yi, H. B. Ahmedov, S. Jiang, Y. Li, S. J. Flinn, and P. G. Fernandes, “Coordinate-Aware Mask R-CNN with Group Normalization: A underwater marine animal instance segmentation framework,” Neurocomputing, vol. 583, p. 127488, May 2024, doi: https://doi.org/10.1016/j.neucom.2024.127488.
M. Kang, S. Kim, K. H. Jin, E. Adeli, K. M. Pohl, and S. H. Park, “FedNN: Federated learning on concept drift data using weight and adaptive group normalizations,” Pattern Recognit., vol. 149, p. 110230, May 2024, doi: https://doi.org/10.1016/j.patcog.2023.110230.
G. A. Kale and U. Yüzgeç, “Advanced strategies on update mechanism of Sine Cosine Optimization Algorithm for feature selection in classification problems,” Eng. Appl. Artif. Intell., vol. 107, p. 104506, Jan. 2022, doi: https://doi.org/10.1016/j.engappai.2021.104506.
E. Raja, B. Soni, C. Lalrempuii, and S. K. Borgohain, “An adaptive cyclical learning rate based hybrid model for Dravidian fake news detection,” Expert Syst. Appl., vol. 241, p. 122768, May 2024, doi: https://doi.org/10.1016/j.eswa.2023.122768.
P. Kumari and P. Saxena, “Cataract detection and visualization based on multi-scale deep features by RINet tuned with cyclic learning rate hyperparameter,” Biomed. Signal Process. Control, vol. 87, p. 105452, Jan. 2024, doi: https://doi.org/10.1016/j.bspc.2023.105452.
B. Sakthivel, P. Vanathi, M. R. Sri, S. Subashini, V. K. Sonthi, and C. Sathish, “Generative AI Models and Capabilities in Cancer Medical Imaging and Applications,” in 2024 3rd International Conference on Sentiment Analysis and Deep Learning (ICSADL), IEEE, Mar. 2024, pp. 349–355. doi: https://doi.org/10.1109/ICSADL61749.2024.00063.
A. Srivastava, B. Pandiya, and N. S. Nautiyal, “Application of Artificial Intelligence in Risk Assessment and Mitigation in Banks,” in Artificial Intelligence for Risk Mitigation in the Financial Industry, Wiley, 2024, pp. 27–52. doi: https://doi.org/10.1002/9781394175574.ch2.
P. Suresh and P. V. Manivannan, “Human driver emulation and cognitive decision making for autonomous cars,” in 2016 International Conference on Robotics: Current Trends and Future Challenges (RCTFC), IEEE, 2016, pp. 1–6. doi: https://doi.org/10.1109/RCTFC.2016.7893411.
G. Macia, A. Liddell, and V. Doyle, “Conversational AI with large language models to increase the uptake of clinical guidance,” Clin. eHealth, vol. 7, pp. 147–152, Dec. 2024, doi: https://doi.org/10.1016/j.ceh.2024.12.001.
W. Ye, C. Liu, Y. Chen, Y. Liu, C. Liu, and H. Zhou, “Multi-style transfer and fusion of image’s regions based on attention mechanism and instance segmentation,” Signal Process. Image Commun., vol. 110, p. 116871, Jan. 2023, doi: https://doi.org/10.1016/j.image.2022.116871.
