Although T cells are recognized as likely important players in SARS-CoV-2 immunity, little is known about the phenotypic features of SARS-CoV-2-specific T cells associated with recovery from COVID-19. We analyzed total and SARS-CoV-2-specific T cells from longitudinal specimens of 34 COVID-19 cases with severities ranging from mild (outpatient) to critical (ICU) culminating in death. The subset distribution of SARS-CoV-2-specific T cells differed in mild vs. severe ICU cases. Longitudinal analyses of severe ICU cases revealed that while those that survived infection mounted an increasing SARS-CoV-2-specific T cell response over time, those that succumbed to infection did not, in a manner associated with antigen-independent T cell activation. Relative to severe cases that culminated in death, individuals that recovered from severe COVID-19 harbored elevated levels of SARS-CoV-2-specific T cells capable of homeostatic proliferation. In the fatal COVID-19 cases, SARS-CoV-2-specific regulatory T cells were elevated and antigen-independent activated CXCR4+ T cells increased over time. Together with the demonstration of elevated proportions of inflammatory CXCR4+ T cells in the lungs of severe COVID-19 patients, these results support a model whereby T cells activated through bystander effects contribute to immunopathology by being non-specifically recruited to the inflamed lung. Overall, these findings suggest that a robust, non-suppressive SARS-CoV-2-specific T cell response is important for limiting immunopathology and recovery from severe COVID-19.