Can Artificial Intelligence Make the Diagnosis? Investigating Automated Psychiatric Case Diagnosis using Large Language Models

Background: In recent years, Large Language Models (LLMs) like ChatGPT have gained prominence in healthcare for interpreting and generating language. Past studies have demonstrated LLMs’ promise to complete clinical knowledge-based tasks, such as completing examination questions. This study explores LLMs’ capacity for psychiatric diagnosis using an iterative model.

Methods: A series of iterative prompts was developed using decision trees from the DSM-5 Handbook of Differential Diagnosis. Ten case prompts were randomly selected from the DSM-5 Clinical Cases handbook. A GPT-4 model generated a list of relevant decision trees for each case prompt. Then, the model was queried iteratively for each tree, resulting in the selection of up to one diagnosis each. The resulting predictions were refined into a list, and then separately aggregated into their associated DSM-5-TR chapter headings. For each case, positive predictive value (PPV, proportion of predicted diagnoses that were correct) and true positive rate (TPR/sensitivity, proportion of correct diagnoses that were predicted) were then calculated at the level of individual diagnoses and chapter headings and then averaged across all cases.

Results: For specific diagnoses, the PPV was 56.7% and the TPR was 35%. At the chapter heading level, the PPV and TPR were 81.7% and 52.7% respectively.

Conclusion: The GPT-4 model demonstrates potential in psychiatric case formulation, with better success in identifying mental disorder categories than specific diagnoses, but does not approach human-level performance. The model often predicted more diagnoses than were appropriate. Further work could improve the model’s ability to narrow down appropriate diagnoses.