An Annotation Model on End-to-End Chest Radiology Reports

Abstract

Annotating radiographic images with tags is an indispensable preliminary work in computer-aided medical research, which requires professional physician participated in and is quite timeconsuming. Therefore, how to automatically annotate radiographic images has become the focus of researchers. However, image report texts, containing crucial radiologic information, have not to be given enough attention for images annotation. In this paper, we propose a neural sequence-to-sequence annotation model. Especially, in the decoding phase, a probability is first learned to copy existing words from report texts or generate new words. Second, to incorporate the patient’s background information, ‘‘indication’’ section of the report is encoded as a sentence embedding, and concatenated with the decoder neural unit input. What’s more, we devise a more reasonable evaluation metric for this annotation task, aiming at assessing the importance of different words. On the Open-i dataset, our model outperforms existing non-neural and neural baselines under the BLEU-4 metrics. To our best knowledge, we are the first to use sequence-to-sequence model for radiographic image annotation.