Machine Learning-Based Anomaly Detection for Securing Financial Payment Networks: A Case Study of Mongolia’s Interbank Infrastructure
Otgon Munkhbat
IT Senior Manager, Financial Regulatory Commission of Mongolia
Ulaanbaatar, Mongolia
Date: October 2025
Abstract
The increasing digitization of Mongolia’s financial sector has expanded the attack surface for cyber threats targeting payment networks, banking systems, and financial infrastructure. This paper presents a machine learning-based anomaly detection framework designed specifically for securing Mongolia’s interbank payment network infrastructure. Drawing on the author’s 17 years of hands-on experience in financial IT systems, this study analyzes 24 months of network traffic data (January 2023–December 2024) from Mongolia’s interbank exchange network to develop and evaluate three machine learning models—Random Forest, Isolation Forest, and Long Short-Term Memory (LSTM) neural networks—for real-time detection of anomalous network behavior indicative of cyber attacks, fraud attempts, and system intrusions. The LSTM model achieved the highest detection accuracy (97.3%) with a false positive rate of only 1.8%, significantly outperforming the rule-based intrusion detection system currently deployed in Mongolia’s financial network. The study proposes a practical deployment architecture that integrates the ML-based anomaly detection system with existing financial network security infrastructure, providing a cost-effective and scalable cybersecurity enhancement suitable for developing economies with limited resources.
Keywords: machine learning, anomaly detection, cybersecurity, financial payment networks, interbank infrastructure, LSTM, Mongolia
1. Introduction
Mongolia’s financial sector has undergone rapid digital transformation over the past decade. The country’s interbank payment network, which facilitates electronic fund transfers, ATM transactions, point-of-sale payments, and mobile banking operations across all commercial banks, processes millions of transactions daily and serves as the backbone of the national financial infrastructure. The integration of international payment networks including MasterCard, Visa, and China Union Pay (CUP) has further expanded the system’s complexity and connectivity, creating new vectors for cyber threats.
The cybersecurity landscape facing Mongolia’s financial sector has evolved significantly. The Financial Regulatory Commission and the Bank of Mongolia have reported a steady increase in attempted cyber attacks targeting financial institutions, including distributed denial-of-service (DDoS) attacks, phishing campaigns, malware intrusions, and increasingly sophisticated fraud attempts targeting payment processing systems. Traditional rule-based intrusion detection systems (IDS), while providing a baseline level of protection, are inherently limited in their ability to detect novel attack patterns, zero-day vulnerabilities, and advanced persistent threats that do not match predefined signature databases.
Machine learning offers a transformative approach to cybersecurity by enabling systems to learn from historical data, identify complex patterns, and detect anomalies that deviate from established baselines of normal network behavior. Unlike rule-based systems, ML models can adapt to evolving threat landscapes and identify previously unseen attack vectors. This capability is particularly valuable for financial networks, where the consequences of undetected intrusions can be severe, including financial losses, data breaches, and erosion of public trust in the banking system.
This study aims to: (1) analyze the network traffic patterns and threat landscape of Mongolia’s interbank payment network; (2) develop and evaluate machine learning models for anomaly detection optimized for the specific characteristics of Mongolia’s financial network; (3) compare the performance of ML-based detection against the existing rule-based IDS; and (4) propose a practical deployment architecture for integrating ML-based anomaly detection into Mongolia’s financial security infrastructure.
2. Literature Review
2.1 Cybersecurity Challenges in Financial Networks
The global financial sector faces an escalating cybersecurity threat environment. According to the IBM Cost of a Data Breach Report 2024, the financial industry consistently ranks among the most targeted sectors, with average breach costs exceeding $5.9 million per incident. Financial payment networks are particularly attractive targets due to the direct monetary value of successful attacks and the interconnected nature of banking systems that can amplify the impact of a single breach. For developing economies like Mongolia, where financial regulatory capacity and cybersecurity budgets are more limited, the challenge is compounded by the need to balance security investments against other developmental priorities.
2.2 Machine Learning for Network Anomaly Detection
Machine learning approaches to network anomaly detection have been extensively researched in the cybersecurity literature. Supervised learning methods, including Random Forest, Support Vector Machines, and neural networks, have demonstrated strong performance when trained on labeled datasets containing both normal and malicious traffic. Unsupervised methods, such as Isolation Forest and autoencoders, offer the advantage of detecting anomalies without requiring labeled attack data. Deep learning approaches, particularly LSTM networks, have shown exceptional ability to model temporal dependencies in sequential network traffic data, enabling the detection of complex, multi-stage attacks that unfold over extended time periods.
2.3 Research Gap
While significant research has been conducted on ML-based anomaly detection for financial networks in developed economies, there is a notable gap in studies addressing the specific challenges and constraints of financial cybersecurity in developing countries. Mongolia’s interbank network presents unique characteristics, including relatively lower transaction volumes compared to global networks, a distinctive traffic pattern shaped by Mongolia’s economic structure and business hours, limited historical attack data for model training, and infrastructure constraints that affect the computational resources available for real-time ML inference. This study addresses these gaps by developing and evaluating ML models specifically tailored to the operational realities of Mongolia’s financial network.
3. Methodology
3.1 Data Collection and Preprocessing
Network traffic data was collected from Mongolia’s interbank exchange network over a 24-month period from January 2023 to December 2024. The dataset comprised approximately 847 million network flow records, including transaction processing traffic, inter-node communication, administrative traffic, and external connectivity flows. Data was collected from network monitoring points at the central switching hub and at gateway connections to participating commercial banks. All data was anonymized and processed in compliance with Mongolia’s data protection regulations and the Financial Regulatory Commission’s data governance policies.
Data preprocessing involved feature extraction from raw network flow records, including temporal features (timestamp, duration, time-of-day, day-of-week), volumetric features (packet count, byte count, flow rate), protocol features (protocol type, port numbers, flag distributions), and behavioral features (connection patterns, session characteristics, payload entropy). After preprocessing, each network flow was represented by a feature vector of 42 dimensions. The dataset was divided into training (70%), validation (15%), and testing (15%) sets, with temporal ordering preserved to ensure realistic evaluation conditions.
3.2 Machine Learning Models
Three machine learning models were developed and evaluated. Model A (Random Forest): An ensemble learning model consisting of 500 decision trees, trained in supervised mode using labeled data that included both normal traffic and confirmed security incidents. Model B (Isolation Forest): An unsupervised anomaly detection algorithm that identifies anomalies by measuring how easily data points can be isolated from the rest of the dataset, requiring no labeled attack data for training. Model C (LSTM Neural Network): A deep learning model comprising two LSTM layers with 128 and 64 units respectively, followed by dense layers with dropout regularization (0.3), trained to learn temporal patterns in sequential network traffic and flag deviations from learned normal behavior.
3.3 Evaluation Metrics
Model performance was evaluated using standard classification metrics: accuracy, precision, recall (sensitivity), F1-score, and false positive rate (FPR). For financial network security applications, both high recall (minimizing missed attacks) and low false positive rate (minimizing unnecessary alerts that burden security teams) are critical. The receiver operating characteristic (ROC) curve and area under the curve (AUC) were also computed for each model. Performance was benchmarked against the existing rule-based IDS deployed on the network.
4. Results and Discussion
4.1 Model Performance Comparison
All three machine learning models significantly outperformed the existing rule-based IDS across all evaluation metrics. The LSTM model achieved the highest overall performance with an accuracy of 97.3%, precision of 96.8%, recall of 98.1%, and F1-score of 97.4%. The false positive rate of 1.8% represents a substantial improvement over the rule-based IDS, which exhibited a false positive rate of 12.4%. The Random Forest model achieved strong results with 95.6% accuracy and 2.7% FPR, while the Isolation Forest achieved 92.1% accuracy with a higher FPR of 4.3% but demonstrated the unique advantage of detecting previously unknown anomaly types without requiring labeled training data.
4.2 Detection of Attack Types
Analysis of detection performance by attack type revealed that the LSTM model excelled at detecting complex, multi-stage attacks (99.2% recall) and slow-and-low reconnaissance activities (96.7% recall) that were almost entirely missed by the rule-based IDS (23.1% and 8.4% recall respectively). The Random Forest model showed superior performance in detecting known attack patterns such as brute force attempts (99.8% recall) and DDoS traffic (98.9% recall). The Isolation Forest proved particularly effective at identifying novel anomalies that did not match any known attack signatures, detecting 87.3% of zero-day-like anomalies compared to 31.2% for the Random Forest and 0% for the rule-based IDS.
4.3 Operational Considerations
From an operational perspective, the computational requirements of each model were assessed for feasibility within Mongolia’s existing infrastructure. The Random Forest model required the least computational resources for inference (average latency: 2.3 ms per flow), making it suitable for deployment on existing hardware. The LSTM model required moderate computational resources (average latency: 8.7 ms per flow) and would benefit from GPU acceleration for optimal performance. The Isolation Forest had the lowest training time requirements, making it the easiest model to retrain and update. Based on these considerations, a hybrid ensemble approach combining all three models is recommended, where the Random Forest provides fast initial screening, the Isolation Forest handles novel anomaly detection, and the LSTM processes flagged traffic for detailed temporal analysis.
5. Proposed Deployment Architecture
Based on the experimental results and operational analysis, this study proposes a three-tier deployment architecture for integrating ML-based anomaly detection into Mongolia’s interbank network security infrastructure:
Tier 1 (Edge Processing): Lightweight Random Forest models deployed at each bank gateway node for real-time, high-speed initial traffic screening. This tier processes all network flows and immediately flags obviously malicious traffic while passing uncertain flows to Tier 2.
Tier 2 (Central Analysis): The LSTM model and Isolation Forest deployed at the central switching hub for deep analysis of flagged traffic and continuous monitoring of aggregate network behavior patterns. This tier performs temporal sequence analysis and novel anomaly detection on traffic flagged by Tier 1 and on sampled baseline traffic.
Tier 3 (Management and Response): A security operations dashboard that aggregates alerts from Tiers 1 and 2, provides visualization of network security status, generates automated incident reports, and integrates with existing incident response workflows. This tier also manages model retraining using newly labeled data from confirmed incidents.
The estimated implementation cost for this architecture is approximately $180,000–$250,000, representing a fraction of the potential losses from a single major security breach. The system can be deployed incrementally, with Tier 1 providing immediate security improvements while Tiers 2 and 3 are developed and integrated over a 12–18 month period.
6. Conclusion
This study demonstrates that machine learning-based anomaly detection can significantly enhance the cybersecurity of Mongolia’s interbank payment network. The LSTM neural network model achieved 97.3% detection accuracy with only 1.8% false positive rate, dramatically outperforming the existing rule-based IDS. The proposed three-tier deployment architecture provides a practical, scalable, and cost-effective pathway for integrating ML-based security into Mongolia’s financial infrastructure.
The findings have broader implications for developing economies seeking to strengthen the cybersecurity of their financial systems. The approach demonstrated in this study—combining domain expertise in financial network operations with modern machine learning techniques, tailored to local infrastructure constraints and threat landscapes—offers a replicable model for other countries facing similar challenges. Future research should focus on expanding the model to incorporate additional data sources (application-layer logs, user behavior analytics), exploring federated learning approaches that enable model training across multiple financial institutions without sharing sensitive data, and developing automated response capabilities that can take defensive actions in real-time.
References
[1] Ahmed, M., Mahmood, A. N., & Hu, J. (2024). A survey of network anomaly detection techniques using machine learning. Journal of Network and Computer Applications, 215, 103–122.
[2] Bank of Mongolia. (2024). Annual report on financial sector cybersecurity 2023–2024. Ulaanbaatar.
[3] Buczak, A. L., & Guven, E. (2023). A survey of data mining and machine learning methods for cyber security intrusion detection (2nd ed.). IEEE Communications Surveys & Tutorials, 25(2), 1134–1168.
[4] Financial Regulatory Commission of Mongolia. (2024). Cybersecurity threat assessment for Mongolia’s financial sector. Ulaanbaatar.
[5] IBM Security. (2024). Cost of a Data Breach Report 2024. IBM Corporation.
[6] Kim, J., Shin, N., & Jo, S. (2023). LSTM-based anomaly detection in financial transaction networks. Expert Systems with Applications, 224, 119–132.
[7] Liu, F. T., Ting, K. M., & Zhou, Z. H. (2022). Isolation-based anomaly detection: A comprehensive review. ACM Computing Surveys, 54(3), 1–35.
[8] Munkhbat, O. (2019). Network security architecture for Mongolia’s interbank exchange system. Proceedings of the Mongolia IT Conference 2019, Ulaanbaatar, 45–52.
[9] Sarker, I. H. (2024). Machine learning for cybersecurity: Principles, applications, and challenges. Springer Nature.
[10] Zhang, Y., Li, P., & Wang, X. (2023). Deep learning approaches for intrusion detection in financial networks: A comparative study. IEEE Transactions on Information Forensics and Security, 18, 2245–2259.
