Leveraging psychological and physiological data has the potential to advance the warfighter and the predictive power of machine learning (ML) to create improved human-computer interactions [1]. This research opportunity aims to collect psychological and physiological data to identify task-relevant decision-making processes. ML model outputs can conflict with user intent, which suggests that critical elements of human decision making are not reflected in the model. The research performed for this fellowship aims to show incorporating decision-making processes into ML models will enhance the corresponding output by improving the ability of the model to predict and respond to human decision-making patterns.

To pinpoint key cognitive processes and corresponding physiological data initially requires identifying attention, memory, and decision-making processes versus task-specific processes. Therefore, experimentation must designed and performed to establish the task-specific processes within applications such geospatial region digitization [2]. During the experimentation, judgements of confidence related to memory and thinking should be collected prior to completing the task. Physiological data such as pupillometry and galvanic skin response should be collected during the task. The collected data will be analyzed to identify processes and to produce cognitive models of task-specific processes. Memory and thinking judgements of confidence should be analyzed to understand patterns of confidence, the calibration of accuracy with confidence, and overall accuracy at the global and individual user level. The cognitive model derived from the confidence data will attempt to explain the relationship between user’s judgements of confidence and user’s decision-making. Pupillometry, behavioral, and physiological data should be analyzed to identify cognitive processes across task difficulty and completion time to understand which metrics account for user performance. The data-derived features representing the cognitive processes should be integrated into the speech-to-text and region digitization task ML models for evaluation. The task ML models shouldthen be evaluated for evidence that incorporating psychological and physiological data improved their output along with their ability to predict and respond to human-decision making patterns.

The accomplishments of this work will be measured by publications in peer-reviewed conferences and journals in addition to patents and technical reports. 

key words

cognitive psychology; interactive machine learning; decision making; machine learning; experimental psychology; human-computer interaction

Citizenship:  Open to U.S. citizens and permanent residents

Level:  Open to Postdoctoral applicants