Seed predation is a key ecosystem process governing the plant population and community structure in forests. Forest fragmentation and habitat loss have been shown to affect seed predation, leading to altered tree recruitment. However, the effects of fragmentation and habitat loss on seed predation are highly variable and context-specific, with limited information from South Asia. For four rainforest tree species in a production landscape of tea and coffee, we examined the influence of forest type (benchmark, N = 3, and fragmented, N = 5, forests) and seed size on the proportion of seeds, 1) predated by vertebrate seed predators, 2) predated by invertebrate seed predators, and 3) removed by vertebrate seed predators. The seed fates were tracked by placing the seeds in four 1 × 1 m plots under the canopy of the tree (N ≥ 11 individuals per tree species and 48 trees overall), and camera traps were placed under a subset of trees (N = 29, trap nights = 712). Overall, we recorded nine species of seed predators, and the seed predator species composition differed for the four tree species. However, there was no significant difference in either seed predator species composition or their visitations across the forest types. The seeds were more likely to be predated (on average 6% higher predation) by vertebrate seed predators in the fragments than in the benchmark forests. Medium-seeded species were more likely to be predated by vertebrate seed predators than large-seeded species (on average 41% higher predation). Invertebrate seed predation and seed removal by vertebrate seed predators had a weak relationship with forest type, with higher invertebrate seed predation in the fragments and higher seed removal in the benchmark forests. With altered seed dispersal patterns resulting from habitat fragmentation and habitat loss, differential seed predation can alter regeneration patterns, thereby influencing adult tree communities in fragmented forests.