SYSTEM CALL TRACE BASED PROBABILISTIC PROGRAM MODELING FOR EXPLOITATION DETECTION

Open Access
Author:
Li, Hao
Graduate Program:
Computer Science and Engineering
Degree:
Master of Science
Document Type:
Master Thesis
Date of Defense:
September 06, 2017
Committee Members:
  • Gang Tan, Thesis Advisor
  • David Jonathan Miller, Thesis Advisor
Keywords:
  • Intusion Detection
  • Program Modeling
  • HMM
  • System Call
Abstract:
Intrusion detection system (IDS) is a common and necessary application for modern software systems to monitor abnormal and potential exploit behaviors. Recent research works have been focusing on anomaly-based IDSs since they have better capabilities of monitoring complex and versatile systems. STatically InitiaLized markOv (STILO) model is one of the recent works that shows superior performance detecting abnormal system call traces. In this thesis, we tested STILO on DARPA CGC final event challenge binaries. Besides that, using STILO method, we built a generic model that works on different software. DARPA CGC is a competition for automatic software defense systems to detect and patch vulnerabilities. Vulnerable challenge binaries are installed on an environment where the competing systems are allowed to analyze and repair it. Those challenge binaries, in our context, are used to test STILO and build generic behavior model upon. The results show that STILO is not able to detect all attacks on DARPA CGC binaries without suffering from high false alarm rates. Similarly, the generic model is not able to perform well, since the model might be confused by the diversified software behavior in training. Another possible reason for the poor performance of STILO is that the attacking requests we used on challenge binaries do not include payloads that usually presents the most distinctive abnormal in behavior.