Enhancing Network Intrusion Detection through Host Clustering

The state-of-the-art in intrusion detection mainly relies on signature-based techniques, which has severe limitations. This research proposes a new approach towards detecting advanced attacks, by focusing on internal network traffic and by using anomaly-based detection. The performance of the anomaly detection is enhanced by using clustering techniques. Internal network traffic is an undervalued source of information for recognising APT-style attacks. Whereas most systems focus on the external border of the network, we show that APT-style campaigns often involve internal network activity. To this end, a framework that shows the relation between attack characteristics and the impact on internal network traffic patterns is presented. To reduce false positive rates and limit the burden of data processing, we propose an additional step in model-based anomaly detection involving host clustering. Through host clustering, individual hosts are grouped together on the basis of their behaviour on the internal network. We argue that a behavioural model for each cluster, compared to a model for each host or a single model for all hosts, performs better in terms of detecting potentially malicious behaviour. We show that by applying this concept to internal network traffic, the detection performance for identifying malicious flows and hosts increases.