Fitness trade-offs across episodes of selection and environments influence life-history evolution and adaptive population divergence. Documenting these trade-offs remains challenging as selection can vary in magnitude and direction through time and space. Here, we evaluate fitness trade-offs at the levels of the whole organism and the quantitative trait locus (QTL) in a multiyear field study of Boechera stricta (Brassicaceae), a genetically tractable mustard native to the Rocky Mountains. Reciprocal local adaptation was pronounced for viability, but not for reproductive components of fitness. Instead, local genomes had a fecundity advantage only in the high latitude garden. By estimating realized selection coefficients from individual-level data on viability and reproductive success and permuting the data to infer significance, we examined the genetic basis of fitness trade-offs. This analytical approach (Conditional Neutrality-Antagonistic Pleiotropy, CNAP) identified genetic trade-offs at a flowering phenology QTL (costs of adaptation) and revealed genetic trade-offs across fitness components (costs of reproduction). These patterns would not have emerged from traditional ANOVA-based QTL mapping. Our analytical framework can be applied to other systems to investigate fitness trade-offs. This task is becoming increasingly important as climate change may alter fitness landscapes, potentially disrupting fitness trade-offs that took many generations to evolve.