Pathogen population dynamics within individual hosts can alter disease epidemics and pathogen evolution, but our understanding of the mechanisms driving within-host dynamics is weak. Mathematical models have provided useful insights, but existing models have only rarely been subjected to rigorous tests, and their reliability is therefore open to question. Most models assume that initial pathogen population sizes are so large that stochastic effects due to small population sizes, so-called demographic stochasticity, are negligible, but whether this assumption is reasonable is unknown. Most models also assume that the dynamic effects of a host’s immune system strongly affect pathogen incubation times or “response times,” but whether such effects are important in real host-pathogen interactions is likewise unknown. Here we use data for a baculovirus of the gypsy moth to test models of within-host pathogen growth. By using Bayesian statistical techniques and formal model-selection procedures, we are able to show that the response time of the gypsy moth virus is strongly affected by both demographic stochasticity and a dynamic response of the host immune system. Our results imply that not all response-time variability can be explained by host and pathogen variability, and that immune system responses to infection may have important effects on population-level disease dynamics.