Pollen fate can strongly affect the genetic structure of populations with restricted gene flow and significant inbreeding risk. We established an experimental population of inbred and outbred Silene latifolia plants to evaluate the effects of (i) inbreeding depression, (ii) phenotypic variation and (iii) relatedness between mates on male fitness under natural pollination. Paternity analysis revealed that outbred males sired significantly more offspring than inbred males. Independently of the effects of inbreeding, male fitness depended on several male traits, including a sexually dimorphic (flower number) and a gametophytic trait (in vitro pollen germination rate). In addition, full-sib matings were less frequent than randomly expected. Thus, inbreeding, phenotype and genetic dissimilarity simultaneously affect male fitness in this animal-pollinated plant. While inbreeding depression might threaten population persistence, the deficiency of effective matings between sibs and the higher fitness of outbred males will reduce its occurrence and counter genetic erosion.