The clinical trajectory of COVID-19 may be influenced by previous responses to heterologous viruses. We examined the relationship of Abs against different viruses to clinical trajectory groups from the National Institutes of Health IMPACC (Immunophenotyping Assessment in a COVID-19 Cohort) study of hospitalized COVID-19 patients. Whereas initial Ab titers to SARS-CoV-2 tended to be higher with increasing severity (excluding fatal disease), those to seasonal coronaviruses trended in the opposite direction. Initial Ab titers to influenza and parainfluenza viruses also tended to be lower with increasing severity. However, no significant relationship was observed for Abs to other viruses, including measles, CMV, EBV, and respiratory syncytial virus. We hypothesize that some individuals may produce lower or less durable Ab responses to respiratory viruses generally (reflected in lower baseline titers in our study), and that this may carry over into poorer outcomes for COVID-19 (despite high initial SARS-CoV-2 titers). We further looked at longitudinal changes in Ab responses to heterologous viruses, but found little change during the course of acute COVID-19 infection. We saw significant trends with age for Ab levels to many of these viruses, but no difference in longitudinal SARS-CoV-2 titers for those with high versus low seasonal coronavirus titers. We detected no difference in longitudinal SARS-CoV-2 titers for CMV seropositive versus seronegative patients, although there was an overrepresentation of CMV seropositives among the IMPACC cohort, compared with expected frequencies in the United States population. Our results both reinforce findings from other studies and suggest (to our knowledge) new relationships between the response to SARS-CoV-2 and Abs to heterologous viruses.