THE EFFECTIVENESS OF CHATGPT IN THE EVALUATION OF ULTRASOUND REPORTS OF PEDIATRIC EMERGENCY PATIENTS

Introduction: This study aims to evaluate the success of the ChatGPT model in detecting emergency pathologies in pediatric emergency ultrasound reports and to compare the effectiveness of zero-shot and few-shot prompting techniques. Material and Method: A total of 196 pediatric emergency ultrasound reports were included in the study. Firstly, the ultrasound reports were anonymized. A board-certified radiologist labeled the reports as positive or negative for emergency pathology. ChatGPT-3.5 and 4 models were tested with different prompting techniques (zero-shot and few-shot) and at different times (March and June). Performance of each model was assessed using accuracy, sensitivity, specificity, and F1-score. The McNemar test was used to compare the binary classes. Results: The most frequently identified emergency pathologies were testicular torsion, appendicitis, and intussusception. ChatGPT-4 (March, zero-shot) achieved the highest success rate (accuracy 94%, sensitivity 91%, specificity 95%). In binary classification, a stat istically significant difference was only found between ChatGPT-3.5 zero-shot (June) and ChatGPT-4 zero-shot (June) responses (p<0.05). The accuracy rates of all models ranged from 88% to 94%. Few-shot prompting did not show a significant performance advantage over the zero-shot technique. Conclusion: ChatGPT models are a successful method for detecting the presence of emergency pathology when compared to the reference standard of radiologist interpretation. Our study demonstrates the potential for large language models to be used as triage tools in clinical workflows.