In recent years, medical educators have been making meaningful attempts to rethink how premedical students are prepared for medical school, and how medical students are prepared for residency. Among the many challenges to redesigning premedical and medical school curricula, one that stands out is the constraint imposed by our current methods of assessing aptitude, particularly our use of the Medical College Admissions Test (MCAT) and the United States Medical Licensing Examination (USMLE). For much of the past century, medical school and residency admissions committees have relied heavily on MCAT and USMLE scores to evaluate and rank candidates to their programs. These high-stakes exams determine to a large extent what is taught, and what is stressed, in preparation for and during medical school - despite the fact that scores have limited ability to predict future success in clinical medicine or biomedical research. Additionally, evidence indicates that students from disadvantaged and minority backgrounds do not fare as well on these exams and, as a result, may be disproportionately excluded from the medical profession. While medical school admissions committees have made limited incremental gains in holistic review, residency programs appear to be increasingly focused on USMLE Step scores and veering away from the spirit of holistic review. The authors propose that substantive change will remain slow in coming unless members of the medical education community radically rethink how we report scores from these exams, and how we use them in our selection of future medical students and residents.