Symbiosis benefits hosts in numerous ways, but much less is known about how host-association affects symbionts. While symbiont fitness can be mediated by host, symbiont, and/or environmental factors, recent works indicate that symbiont performance can depend on whether the symbiont is needed by the host, suggesting that symbiosis is not always beneficial for symbionts. To determine the impact of symbiosis on symbionts across the Tree of Life, we conducted a meta-analysis across 83 unique host-symbiont pairings under a range of spatial and temporal contexts. Specifically, we asked how symbiont fitness is altered outside of symbiosis, when host-symbiont interaction is under suboptimal conditions, or as hosts age. We found that intracellular symbionts associated with protists tend to have greater fitness outside of symbiosis, with the opposite trend for animal hosts. This result suggests that animals may be better at maintaining symbionts. Symbiont fitness also generally increased as hosts grow older. Moreover, symbionts that can proliferate in- and outside host cells performed better than those found exclusively inside or outside cells, suggesting that flexibility in growing location may help symbionts thrive. We discuss these fitness patterns in light of host-driven factors, where hosts exert influence over symbionts to suit their needs.