Castles Built on Sand: Observations from Classifying Academic Cybersecurity Datasets with Minimalist Methods

Abstract

Machine learning (ML) has been a staple of academic research into pattern recognition in many fields, including cybersecurity. The momentum of ML continues to speed up alongside the advances in hardware capabilities and the methods they unlock, primarily (deep) neural networks. However, this article aims to demonstrate that the non-judicious use of ML in two prominent domains of data-based cybersecurity consistently misleads researchers into believing that their proposed methods constitute actual improvements. Armed with 17 state-of-the-art datasets in traffic and malware classification and the simplest possible machine learning model this article will show that the lack of variability in most of these datasets immediately leads to excellent models, even if that model is only one comparison per feature.