Research on phenotypic plasticity has often focused on how a given genotype responds to changing physical environments. However, for many species the social environment plays an equally important role due to competition for resources. During early development, the level of competition for limited resources will often depend critically on the number of siblings. Therefore, competition among siblings should drive the evolution of genes that allow flexible responses to realized levels of competition and maternal resource availability. However, it is unknown whether genetically based differences between individuals exist in their response to the social environment that affects their future development. Using a quantitative trait locus approach in an experimental population of mice we demonstrate that effects of sibling number on body weight depend on individual genotype at seven loci, over and above the general negative litter size effect. Overall, these litter size-by-genotype interactions considerably modified the degree to which increasing litter size caused reduced weight. For example at one locus this effect leads to a 7% difference in body weight at week 7 between individuals experiencing the extremes of the normal range of litter sizes in our population (five to nine). The observed interaction between genotype and the competitive environment can produce differences in body weight that are similar in magnitude to the main effect of litter size on weight. Our results show that different genotypes respond to the social environment differentially and that interaction effects of genotype with litter size can be as important as genotype-independent effects of litter size.