Lilium tsingtauense Gilg is a rare and endangered wild plant, but there is insufficient research on the environmental drivers of intraspecific variation in its seed and fruit traits. To investigate the responses of variations in seed and fruit traits to geographical and soil factors across different habitats, 37 sample plots were selected for investigation and statistics within an elevation range of 200–1000 m in Laoshan, China. Mature fruit and soil samples were brought back to measure soil nutrient content, fruit size, seed number, and seed germination rate. The results showed that: (a) There are differences in geographical and soil factors among the habitats of different L. tsingtauense populations. There were significant differences in elevation, aspect, light intensity, soil water content, soil electrical conductivity, soil organic matter content, and soil total nitrogen content among different populations. (b) Fruiting ability responds more readily to environmental changes than fruit and seed traits do. Fruit length, width, and thousand-grain weight were more stable than the number of plump seeds per fruit and germination percentage. (c) Significant positive correlations were observed between longitude, elevation, light intensity, soil water content, soil electrical conductivity and fruit and seed traits, while soil total phosphorus content showed a significant negative correlation with fruit and seed traits. Among these, elevation was identified as a potential key environmental factor driving variations in fruit and seed traits of L. tsingtauense. Individuals growing at higher elevations exhibited greater fruit production and higher seed germination rates. These findings reveal the environmental variability in fruit and seed traits of L. tsingtauense and its influencing factors, providing important insights for identifying core conservation areas and guiding habitat restoration for this wild resource.