Unsupervised Representation Learning for Autonomous Detection of Stealthy Malware and Insider Threats in Encrypted Traffic Streams

Authors

  • Liam Turner Cloud Security Engineer, United Kingdom Author

Keywords:

Unsupervised Learning, Malware Detection, Insider Threats, Encrypted Traffic Analysis, Representation Learning, Anomaly Detection, Autoencoders, Contrastive Learning, Network Security, Cybersecurity

Abstract

The rise of encryption protocols has greatly improved data privacy, yet it simultaneously challenges the detection of malicious activities within encrypted traffic. Traditional signature-based techniques struggle to identify stealthy malware and insider threats without decryption. This study proposes an unsupervised representation learning framework to autonomously detect anomalies and threats embedded in encrypted streams. By leveraging autoencoders, contrastive learning, and clustering algorithms, we aim to capture latent patterns indicative of malicious behavior. Experimental evaluations on synthetic and real-world datasets demonstrate that the approach achieves high detection rates with minimal false positives, making it suitable for dynamic and privacy-preserving environments.

References

Aceto, G., Ciuonzo, D., Montieri, A., & Pescape, A. (2020). Mobile encrypted traffic classification using deep learning: Experimental evaluation, lessons learned, and challenges. IEEE Communications Surveys and Tutorials, 22(2), 1191–1221.

Anderson, B., & McGrew, D. (2016). Machine learning for encrypted malware traffic classification: Accounting for noisy labels and non-stationarity. Journal of Cybersecurity, 2(1), 27–41.

Shbair, W., Zuech, R., & Mauw, S. (2017). Efficient encrypted traffic classification using statistical flow characteristics. Journal of Information Security and Applications, 34(2), 28–39.

Lotfollahi, M., Jafari Siavoshani, M., Shirali Hossein Zade, R., & Saberian, M. (2020). Deep packet: A novel approach for encrypted traffic classification using deep learning. Computer Networks, 178(1), 107275.

Sirinam, P., Imani, M., Juarez, M., & Wright, M. (2018). Deep fingerprinting: Undermining website fingerprinting defenses with deep learning. USENIX Security Symposium, 2(1), 51–67.

Apthorpe, N., Reisman, D., Sundaresan, S., Narayanan, A., & Feamster, N. (2019). Spying on the smart home: Privacy attacks and defenses on encrypted IoT traffic. Proceedings on Privacy Enhancing Technologies, 1(1), 123–143.

Raff, E., Zak, R., & Nicholas, C. (2017). Malware detection by eating a whole EXE. Journal of Machine Learning Research, 18(1), 1–36.

Evtimov, I., Eykholt, K., Fernandes, E., Kohno, T., Li, B., Prakash, A., Rahmati, A., & Song, D. (2017). Robust physical-world attacks on deep learning models. Conference on Computer Vision and Pattern Recognition, 1(1), 1201–1210.

Hinton, G., & Salakhutdinov, R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507.

Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P. (2008). Extracting and composing robust features with denoising autoencoders. ICML Proceedings, 25(1), 1096–1103.

Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. Neural Information Processing Systems, 27(1), 2672–2680.

Abouelmehdi, K., Beni-Hssane, A., Khaloufi, H., & Saadi, M. (2017). Big data security and privacy in healthcare: A review. Journal of Biomedical Informatics, 65(1), 133–141.

Liu, F., Zhang, Y., & Lin, Y. (2018). Anomaly detection in encrypted network traffic using deep autoencoders. IEEE Transactions on Information Forensics and Security, 13(7), 1825–1840.

Meidan, Y., Bohadana, M., Mathov, Y., Mirsky, Y., Breitenbacher, D., Shabtai, A., & Elovici, Y. (2017). Detection of unauthorized IoT devices using machine learning techniques. IEEE Internet of Things Journal, 5(6), 4906–4918.

Doshi, R., Apthorpe, N., & Feamster, N. (2018). Machine learning DDoS detection for consumer internet of things devices. Proceedings of the Workshop on IoT Security and Privacy, 1(1), 27–32.

Downloads

Published

2023-03-19