Virtualization-based Security Analysis of Production Server Systems

Open Access
Zhang, Shengzhi
Graduate Program:
Computer Science and Engineering
Doctor of Philosophy
Document Type:
Date of Defense:
July 18, 2012
Committee Members:
  • Peng Liu, Dissertation Advisor
  • Sencun Zhu, Committee Member
  • Bhuvan Urgaonkar, Committee Member
  • Dr Soundar Kumara, Committee Member
  • Virtualization
  • security analysis
  • driver diversity
  • heterogeneous virtual machine migration
  • availability preserving
Production server systems are critical resources in the era of network-centric warfare. With the rapid prevalence of E-Commerce, on-line gaming, social networking, logistics, and Cloud Computing, the demand on service continuity and availability is increasingly crucial to production servers or data centres. Any downtime or malfunction caused by vulnerability exploitation leads to productivity and profit loss. For instance, drivers (accounting for more than half of most commodity operating system kernels), especially third party drivers, could contain malicious code (e.g., logic bombs) and/or carefully designed-in vulnerabilities. Once got executed/exploited, such compromised drivers render the attackers the opportunity of leveraging drivers’ privilege to interrupt the intend-to-guard services. Hence, production server systems need undergo comprehensive security analysis to be sufficiently resistant to attacks to guarantee continuous service and correct execution. In this dissertation, I propose a set of automatic security analysis mechanisms to help commodity systems to be resilient to vulnerability exploitation. These mechanisms finally help server systems automatically preserve service continuity and correct execution upon vulnerability exploitation. Specifically, an intrusion harm analysis system has been built to comprehensively evaluate the damage caused by attacks to the production server systems. It allows “imperfect” or vulnerable software to be deployed in trustworthy enterprise production environment. By providing automatic checkpointing and intrusion analysis however, the protected systems can obtain precise knowledge of the damage that has been caused, which would enable the systems to do appropriate availability-integrity tradeoff in generating the recovery plan. As the significant increase of vulnerable drivers, a trustworthiness assessment of third party drivers is also proposed against kernel integrity manipulation, confidentiality tampering, and resource abuse. Then, only the outweighing drivers can be deployed in trustworthy production environment with the proposed operate-through system preserving the states of critical service applications against vulnerability exploitation. First, I propose PEDA (Production Environment Damage Analysis) system to comprehensively analyze the harm of intrusion to production servers, by decoupling the onerous analysis work from the on-line execution. Once the system being compromised, the “has-been-infected” execution is analyzed during high fidelity replay on a separate instrumentation platform. The replay is implemented based on the heterogeneous virtual machine migration. The servers’ on-line execution runs atop fast hardware-assisted virtual machines (such as Xen for near native speed execution), while the infected execution is replayed atop binary instrumentation virtual machines (such as QEMU for instrumentation platform). From identified intrusion symptoms, PEDA is capable of locating the fine-grained taint seed by integrating the backward system call dependency tracking and one-step-forward taint information flow auditing. Started with the fine-grained taint seed, PEDA applies dynamic taint analysis during the replayed execution to provide the most fine-grained intrusion harm analysis. Second, I present a novel driver evaluation approach, Heter-device, to fully analyze drivers’ trustworthiness before putting any trust on them. Heter-device relies on virtual platforms to emulate heterogeneous device (Heter-device) pairs (e.g., Intel 82540EM NIC and Realtek RTL8139) for guest operating system replicas. Each replica loads heterogeneous drivers corresponding to the devices it runs on. Heter-device approach stands on the assumption that heterogeneous drivers should not have the same exploitable vulnerability due to their separate developing procedures. Thus they provide an implicit and complete reference model for each other when trustworthiness assessment is conducted via fine-grained auditing. By deploying Heter-device as a high-interaction honeypot with the synchronization points and monitoring “sensors”, I can closely compare the divergence of two replicas when the vulnerable driver is being compromised. Hence, multiple attack vectors of compromised drivers, including kernel integrity manipulation, resource abuse, and confidentiality tampering can be faithfully revealed. Last, I present DRASP (Diverse Replica based Application State Preserving), a mechanism that leverages Heter-device architecture to help protected systems operate-through vulnerability exploitation. Once the proposed virtualized device diversity is deployed, the system replicas need to load different drivers as loadable kernel modules for the diverse devices. Again, the idea is that different drivers should have different vulnerabilities, in terms of the location or the details of the vulnerability. Hence, one driver vulnerability exploitation can at most succeed in one replica, with the other replica surviving. Once the successful exploitation tampers the applications’ code/data or compromises the applications’ metadata for commercial benefits, it can be detected through the proposed response or state validation. Afterwards, the application on the survival replica can immediately take over the workload to preserve the service continuity and accumulated state, while ensuring the correct execution.