COMPARISONS OF MEASURES OF CRITICAL THINKING AND ANALYTIC PERFORMANCE: EXPLORATORY STUDIES

Restricted (Penn State Only)
Author:
Hodgson, John Gregory
Graduate Program:
Information Sciences and Technology
Degree:
Doctor of Philosophy
Document Type:
Dissertation
Date of Defense:
December 07, 2018
Committee Members:
  • Steven Raymond Haynes, Dissertation Advisor
  • Steven Raymond Haynes, Committee Chair
  • Donald Richard Shemanski, Committee Member
  • Edward J Glantz, Committee Member
  • Guido Cervone, Outside Member
Keywords:
  • critical thinking
  • intelligence analysis
  • participatory action research
Abstract:
There is a substantial body of scholarship on critical thinking in the fields of philosophy, education, and cognitive psychology. Several mechanisms exist to measure critical thinking skills, abilities, and dispositions, with high measures of reliability and validity. Exploration of critical thinking specific to intelligence analysis has expanded significantly since 2001 and has become a focused area of inquiry. However, most researchers in this sub-field equate “critical thinking” with structured analytic methods and largely ignore the complex range of cognitive dimensions that scholars in other domains embrace. Researchers and other authors in intelligence analysis posit a relationship between critical thinking education and the quality of analytic performance, but findings to date are mixed. Researchers have not yet explored standard assessment measures of critical thinking as predictive mechanisms to determine which analysts may demonstrate higher or lower levels of job performance. Furthermore, scholarship in critical thinking for intelligence analysis lacks consistency regarding definitions as well as the understanding and articulation of the task environment, including analytic processes, products, and evaluation. Within the community of practice, intelligence organizations continue to face strong public and political pressure to reform in the wake of several perceived intelligence failures in recent years, such as a lack of warning regarding the terrorist attacks on New York and Washington, D.C., London, Mumbai, and more recently, Paris, Belgium, Boston, and Germany. U.S. Intelligence Community reform efforts have been comprehensive, including modifications to organizational structures, increased inter-agency coordination, investment in education and training programs, and emphasis on analytic processes and products. The need for improved research and practice in critical thinking has emerged as a common theme regarding those reform efforts. These requirements are even more critical because of the continued expansion of the U.S. national security enterprise and the significant personnel changeover that continues as the “baby boom” generation retires from the workforce. The demand for skilled entry-level analysts remains high. To address these challenges, this study explores the relationship between measures of critical thinking abilities and the quality of analytic processes and products. Specifically, I seek to explore whether standard assessment measures of critical thinking predict the performance of novices in intelligence analysis tasks. Participatory action research provides an appropriate research approach given the requirement for domain expertise and the lack of relevant research. Synthesis of prior work and cognitive models yield an integrated framework for critical thinking in intelligence analysis. I conduct two exploratory studies with multiple methods. First, I conduct a qualitative research project to understand and summarize how practitioners define and describe the dimensions of analytic quality. Specific methods include participatory observation, interviews, and expert panels. This study yields an operationalized model and rubric for analytic quality in the context of intelligence analysis. The second study explores the relationship between standard assessment mechanisms of critical thinking and the performance of individual novices on intelligence analysis tasks. Statistical analysis reveals correlations between some elements of critical thinking and analytic performance. Specifically, certain dispositions for critical thinking may predict analytic performance. Reflection on the three components of this study yields recommendations for future work, implications for the exploration of critical thinking for intelligence analysis, and implications for broader research efforts.