The detrimental effects of matings between relatives are well known. However, few studies determine the extent to which inbreeding depression in males is due to natural or sexual selection. Importantly, measuring fitness or key fitness components, rather than phenotypic traits allows more accurate estimation of inbreeding depression. We investigate how differences in inbreeding and juvenile diet (i.e. early stressful environment) influence a key component of male fitness, namely their reproductive success. We experimentally created inbred and outbred male mosquitofish (Gambusia holbrooki) by mating full-sibs (f = 0·25). We show that this led to a 23% reduction in genome-wide heterozygosity based on SNPs. Males were raised on different diets early in life to create high-stress and low-stress rearing environments. We then allowed adult males to compete freely for females to test if inbreeding, early diet and their interaction affect a male's share of paternity. Early diet had no effect on paternity, but outbred males sired almost twice as many offspring as inbred males (n = 628 offspring from 122 potential sires). Using artificial insemination methods we determined that this was unlikely to be due to early embryo mortality of eggs fertilised by inbred males: there was no evidence that male inbreeding status affects the realised fecundity of females (n = 288). Given there was no difference in male mortality in our competitive mating experiment, the lower reproductive success of inbred males can most parsimoniously be attributed to inbreeding negatively affecting sexually selected traits that affect male mating success and/or sperm competitiveness. We discuss which sexually selected traits might be involved.