The impact of species loss from competitive communities partly depends on how populations of the surviving species respond. Predicting the response should be straightforward using models that describe population growth as a function of competitor densities; but these models require accurate estimates of interaction strengths. Here, we quantified how well we could predict responses to competitor removal in a community of annual plants, using a combination of observation and experiment. It was straightforward to fit models to multi-species communities, which passed standard diagnostic tests and provided apparently sensible estimates of interaction strengths. However, the models consistently underpredicted the response to competitor removal, by a factor of at least 50%. We argue that this poor predictive ability is likely to be general in plant communities due to ‘the ghost of competition present’ that confines species to parts of the environment in which they compete best.