Medical Text and Image Reasoning for Radiation Oncology via Large Language Model

Evaluating text based logical reasoning and medical image reasoning abilities in large language models (LLMs) within the domain of radiation oncology is often challenging. Testing LLMs without fine-tuning in this context often fails to achieve high accuracy, particularly when dealing with domain-specific questions, limiting their utility in medical decision-making. In this paper, five prominent LLMs - ChatGPT-4o, Claude-3.5 Sonnet, Llama 3.2, Gemini 1.5 Flash, and o1-preview are evaluated to assess their text based logical reasoning and medical image reasoning performances on two publicly available datasets. We propose a new challenging benchmark dataset featuring a mixture of single-choice questions as well as multiple choice questions in radiation oncology for the reasoning task. To overcome the limitation of LLMs in handling complex and specialized tasks, we also propose a new hybrid fine-tuning method, where Llama 3.2 is fine-tuned using QLoRA, combined with prompt engineering technique to enhance its reasoning performance. Extensive experiments demonstrate that o1-preview outperforms the other models in overall accuracy, while the fine-tuned Llama 3.2 using our proposed method achieves improved performance, showcasing the effectiveness of hybrid fine-tuning for enhancing LLMs reasoning performance.