Highly frequent pollination at short distances combined with spatial genetic structure (SGS) among reproductive individuals, as observed in populations of many forest tree species, may lead to highly frequent pollinations between genetically related individuals located close to each other. This could increase frequencies of biparental inbreeding and strengthen the degree of SGS across generations. However, the degree of SGS observed in forest tree populations is not usually strong, suggesting that such processes are restricted by other factors, which may include inbreeding depression. To test this hypothesis, we examined inbreeding depression in Castanopsis sieboldii saplings in a population known to have significant SGS and high frequencies of short-distance pollination. We determined genotypes at eight microsatellite loci of 125 adult trees and 899 saplings in a 4-ha plot. The neighborhood model approach, assuming an exponential power function, showed highly frequent pollination at short distance among parents of saplings. SGS was significantly stronger among saplings than among adults. The average value of F is (individual inbreeding coefficient based on kinship coefficients between genes within individuals) was also significantly higher among saplings than among adults. Furthermore, average values for saplings were relatively high (0.036–0.052) in 30–50 to 90–110 cm height classes and rapidly decreased to 0.006 for the ≥110-cm class. Analysis of covariance, taking into account the canopy state, showed that inbreeding depression had a significant negative effect on the saplings’ height. Thus, the level of inbreeding in saplings decreased with size classes increasing, probably due to pronounced inbreeding depression resulting in mortality or reduced growth of inbred saplings, and hence, the inbreeding depression may prevent increasing of the level of inbreeding and the degree of SGS in subsequent generations.