Dispersal as well as population growth is a key demographic process that determines population dynamics. However, determining the effects of environmental covariates on dispersal from spatial-temporal abundance proxy data is challenging owing to the complexity of model specification for directional dispersal permeability and the extremely high computational loads for numerical integration. In this paper, we present a case study estimating how environmental covariates affect the dispersal of Japanese sika deer by developing a spatially explicit state-space matrix model coupled with an improved numerical integration technique (Markov chain Monte Carlo with particle filters). In particular, we explored the environmental drivers of inhomogeneous range expansion, characteristic of animals with short dispersal. Our model framework successfully reproduced the complex population dynamics of sika deer, including rapid changes in densely populated areas and distribution fronts within a decade. Furthermore, our results revealed that the inhomogeneous range expansion of sika deer seemed to be primarily caused by the dispersal process (i.e., movement barriers in fragmented forests) rather than population growth. Our state-space matrix model enables the inference of population dynamics for a broad range of organisms, even those with low dispersal ability, in heterogeneous landscapes, and could address many pressing issues in conservation biology and ecosystem management.