Individual differences in male reproductive success drive genetic drift and natural selection, altering genetic variation and phenotypic trait distributions in future generations. Therefore, identifying the determinants of reproductive success is important for understanding the ecology and evolution of plants. Here, based on the spatially-explicit mating model (the neighborhood model), we develop a hierarchical probability model that links co-dominant genotypes of offspring and candidate parents with the phenotypic determinants of male reproductive success. The model accounts for pollen dispersal, genotyping errors as well as individual variation in selfing, pollen immigration, and differentiation of immigrant pollen pools. Unlike the classic neighborhood model approach, our approach is specially designed to account for excessive variation (overdispersion) in male reproductive success. Using the Bayesian framework, we implemented the estimation method that, among others, allows the selection of phenotypic variables important for male reproductive success as well as the computation of the fraction of total variance in fecundity (R²) explained by selected variables. Computer simulations showed that our method outperforms both the classic neighborhood model and the two-step approach, where individual fecundities and the effects of phenotypic variables are estimated in separate steps. The analysis of two data examples showed that in wind-pollinated trees, male reproductive success depends on both the amount of produced pollen and the ability to pollen spread. However, despite that the tree size was positively correlated with male fecundity, it explained only a fraction of the total variance in reproductive success, indicating the presence of significant unmeasured factors. Finally, case studies highlighted the importance of accounting for pollen dispersal in the estimation of determinants of plant reproductive success.

Tree coordinates (XY) are given in meters. Microsatellite allele lengths are given in base pairs. Phenotypic measurements for trees are given as standardized values (mean=0 and standard deviation=1). Trees that can serve as pollen parents (hemraphrodites or male trees) are denoted with the variable "Pollen donor"=1.

Missing values are coded with NA.