We present a mechanistic mathematical model of immune checkpoint inhibitor therapy to address the oncological need for early, broadly applicable readouts (biomarkers) of patient response to immunotherapy. The model is built upon the complex biological and physical interactions between the immune system and cancer, and is informed using only standard-of-care CT. We have retrospectively applied the model to 245 patients from multiple clinical trials treated with anti-CTLA-4 or anti-PD-1/PD-L1 antibodies. We found that model parameters distinctly identified patients with common (n = 18) and rare (n = 10) malignancy types who benefited and did not benefit from these monotherapies with accuracy as high as 88% at first restaging (median 53 days). Further, the parameters successfully differentiated pseudo-progression from true progression, providing previously unidentified insights into the unique biophysical characteristics of pseudo-progression. Our mathematical model offers a clinically relevant tool for personalized oncology and for engineering immunotherapy regimens.