While patterns in species diversity have been well studied across large-scale environmental gradients, little is known about how species' interaction networks change in response to abiotic and biotic factors across such gradients. Here we studied seed-dispersal networks on 50 study plots distributed over ten different habitat types on the southern slopes of Mt. Kilimanjaro, Tanzania, to disentangle the effects of climate, habitat structure, fruit diversity and fruit availability on different measures of interaction diversity. We used direct observations to record the interactions of frugivorous birds and mammals with fleshy-fruited plants and recorded climatic conditions, habitat structure, fruit diversity and availability. We found that Shannon interaction diversity (H) increased with fruit diversity and availability, whereas interaction evenness (EH) and network specialization (H2') responded differently to changes in fruit availability depending on habitat structure. The direction of the effects of fruit availability on EH and H2' differed between open habitats at the mountain base and structurally complex habitats in the forest belt. Our findings illustrate that interaction networks react differently to changes in environmental conditions in different ecosystems. Hence, our findings demonstrate that future projections of network structure and associated ecosystem functions need to account for habitat differences among ecosystems.