Gene drive technology, in which fast-spreading engineered drive alleles are introduced into wild populations, represents a promising new tool in the fight against vector-borne diseases, agricultural pests, and invasive species. Due to the risks involved, gene drives have so far only been tested in laboratory settings while their population-level behavior is mainly studied using mathematical and computational models. The spread of a gene drive is a rapid evolutionary process that occurs over timescales similar to many ecological processes. This can potentially generate strong eco-evolutionary feedback that could profoundly affect the dynamics and outcome of a gene drive release. We therefore argue for the importance of incorporating ecological features into gene drive models. We describe the key ecological features that could affect gene drive behavior, such as population structure, life-history, environmental variation, and mode of selection. We review previous gene drive modeling efforts and identify areas where further research is needed. As gene drive technology approaches the level of field experimentation, it is crucial to evaluate gene drive dynamics, potential outcomes, and risks realistically by including ecological processes.