Preserving System Integrity in Commodity Computers

Open Access
Xiong, Xi
Graduate Program:
Computer Science and Engineering
Doctor of Philosophy
Document Type:
Date of Defense:
October 01, 2012
Committee Members:
  • Peng Liu, Dissertation Advisor
  • Sencun Zhu, Committee Member
  • Trent Ray Jaeger, Committee Member
  • David Miller, Committee Member
  • Software Security
  • Operating System Security
  • Virtualization
  • Intrusion Recovery
  • OS Kernel
Today people rely more and more on commodity computer systems for storing and processing information. To make computer systems more trustworthy, it is highly demanding that these systems could have integrity protection mechanism as the security basis of computing. In this dissertation, we propose proactive and reactive approaches to preserve system integrity for commodity computer systems. First, we explore reactive techniques to recover OS-level objects (e.g., processes and files) in an intruded computer system which already has integrity compromise. We design and implement SHELF, an intrusion recovery system that aims to preserve business continuity, availability and recovery accuracy. SHELF tracks activities of a computer system so that it can precisely determine which object of the system is compromised upon given an infection symptom. During the recovery phase, SHELF preserves accumulated clean state of infected objects, and it helps benign objects maintain their availability level to reduce system downtime. The effort of repairing OS-level applications and files, however, must depends on a trusted and uncompromised OS kernel to provide correct functionality and abstractions. As commodity OS kernels are more and more becoming favorable targets for attackers, it is necessary to have proactive protection mechanism to secure the OS kernel and provide solid foundation for use-space security approaches. We study the problem of securing untrusted code executing in the kernel space, which is the major venue for OS kernel integrity compromise. We design and implement HUKO, a hypervisor-based integrity protection system that protects commodity OS kernels from untrusted extensions. In HUKO, untrusted extensions can safely run in the kernel space to provide desired functionality, but they are also confined by access control mechanisms, which significantly limit the attacker's ability to compromise the integrity of OS kernel. Based on the hypervisor architecture provided by HUKO, we further propose SILVER, a comprehensive framework that offers transparent protection domain primitives to achieve fine-grained access control and secure communication between programs in OS kernel. SILVER provides OS kernel developers the ability to specify security properties of their own code and data at the granularity of individual functions and data objects. Moreover, SIVER helps developers to prevent attacks exploiting kernel program communication, which cannot be effectively handled by typical isolation systems. To achieve such mechanism, we propose a novel resource management scheme of kernel data objects according to their security properties. Based on this organization, SILVER enforces access control and communication safety using hypervisor-based memory protection and run-time checks.