Biological invasions represent both a pressing environmental challenge and an opportunity to investigate fundamental ecological processes, such as the role of top predators in regulating species diversity and food-web structure. In whole-ecosystem manipulations of small Caribbean islands where brown anole lizards (Anolis sagrei) were the native top predator, we experimentally staged invasions by competitors (green anoles, A. smaragdinus) and/or novel top predators (curly-tailed lizards, Leiocephalus carinatus). We show that curly-tails destabilized coexistence of competing prey species, contrary to the classic idea of keystone predation. Fear-driven avoidance of predators collapsed the spatial and dietary niche structure that otherwise stabilized coexistence, intensifying interspecific competition within predator-free refuges and contributing to green-anole population extinctions. Moreover, whereas adding either green anoles or curly-tails lengthened food chains, adding both species reversed this effect, in part because apex predators were trophic omnivores. Our results underscore the importance of top-down control in community ecology, but show that its outcomes hinge on prey behavior, spatial structure, and omnivory. Diversity-enhancing effects of top predators cannot be assumed, and non-consumptive effects of predation risk may be a widespread constraint on species coexistence.