Packet Inspection for Application Classification and Intrusion Detection

Open Access
Author:
Wang, Jisheng
Graduate Program:
Electrical Engineering
Degree:
Doctor of Philosophy
Document Type:
Dissertation
Date of Defense:
February 28, 2008
Committee Members:
  • David Jonathan Miller, Committee Chair/Co-Chair
  • George Kesidis, Committee Chair/Co-Chair
  • Nirmal K Bose, Committee Member
  • Prasenjit Mitra, Committee Member
Keywords:
  • Polymorphic Worms
  • Voice over IP
  • Data Digesting
  • Application Classification
  • Intrusion Detection System
  • Network Anomaly Detection
Abstract:
Current computer networks remain vulnerable to a variety of families of attacks including scanning worms, distributed denial-of-service (DDoS) attacks targeting resources associated with end-systems or critical network protocols, and hit-list worms. These kinds of attacks remain significant direct and indirect threats to the network’s infrastructure and its end-systems. Despite past developments, anomaly detection and response targeting zero-day attacks (as not yet seen) remains an open research problem. This dissertation presents the complete structure of an automated payload-based network intrusion detection system, which includes three main components: network traffic mining, network anomaly identification, and worm signature extraction. Estan et al.’s multidimensional digesting algorithm is introduced to mine significant flows – either worm flows or dominant normal flows – among entire network traffic, and several techniques are proposed for improving its efficiency. Based on the mining results, a new entropy-based criterion is presented to correctly identify anomaly network traffic, including the Slammer and Code-Red worms and the DDoS attacks. Moreover, a Generalized Suffix Tree-based approach is proposed for efficiently extracting signatures of polymorphic worms. Therefore, the proposed intrusion detection system can automatically generate signatures of zero-day attacks/worms which can be used to contain their spread in the future. Meanwhile, with the increasing flexibility in current networks, tons of new applications appear and begin to dominate the Internet. The newly emerging peer-to-peer applications, such as Bitcomet and Skype, can be responsible for more than 80% of the total traffic volume in the Internet. Therefore, it is essential for Internet service providers to correctly identify these new applications. This dissertation presents an efficient approach to identify Skype voice over IP (VoIP) traffic by using reliable statistical information. Because of its efficiency in both computational complexity and memory consumption, the new approach can be implemented on network backbone routers to identify Skype VoIP traffic in real-time.