Open Access, Peer-reviewed

ISSN 2005-7571 (Online)
Volume 27, Number 1 (1/2020)
Original Article <page. 18-26 >
DOI : 10.22857/kjbp.2020.27.1.003

Application of Text-Classification Based Machine Learning in Predicting Psychiatric Diagnosis

Doohyun Pak, MD;Mingyu Hwang, MD;Minji Lee, MD;Sung-Il Woo, MD;Sang-Woo Hahn, MD;Yeon Jung Lee, MD; and Jaeuk Hwang, MD

Department of Psychiatry, Soonchunhyang University Seoul Hospital, Seoul, Korea

Objectives : The aim was to find effective vectorization and classification models to predict a psychiatric diagnosis from text-based medical records.

Methods : Electronic medical records (n = 494) of present illness were collected retrospectively in inpatient admission notes with three diagnoses of major depressive disorder, type 1 bipolar disorder, and schizophrenia. Data were split into 400 training data and 94 independent validation data. Data were vectorized by two different models such as term frequency-inverse document frequency (TF-IDF) and Doc2vec. Machine learning models for classification including stochastic gradient descent, logistic regression, support vector classification, and deep learning (DL) were applied to predict three psychiatric diagnoses. Five-fold cross-validation was used to find an effective model. Metrics such as accuracy, precision, recall, and F1-score were measured for comparison between the models.

Results : Five-fold cross-validation in training data showed DL model with Doc2vec was the most effective model to predict the diagnosis (accuracy = 0.87, F1-score = 0.87). However, these metrics have been reduced in independent test data set with final working DL models (accuracy = 0.79, F1-score = 0.79), while the model of logistic regression and support vector machine with Doc2vec showed slightly better performance (accuracy = 0.80, F1-score = 0.80) than the DL models with Doc2vec and others with TF-IDF.

Conclusions : The current results suggest that the vectorization may have more impact on the performance of classification than the machine learning model. However, data set had a number of limitations including small sample size, imbalance among the category, and its generalizability. With this regard, the need for research with multi-sites and large samples is suggested to improve the machine learning models.

Key words : Text-classification;Electronic medical record;Vectorization;Machine learning;Present illness;Psychiatric diagnosis.


This Article