A population’s potential for rapid evolutionary adaptation can be estimated from the amount of genetic variation in fitness-related traits. Inshore populations of the mustard hill coral (Porites astreoides) have been shown to be more tolerant to thermal stress than offshore populations, but it is unclear whether this difference is due to long-term physiological acclimatization or genetic adaptation. Here, we evaluated variation in growth rate and survival among 38 families of juvenile recruits of P. astreoides spawned by colonies originating from inshore and offshore locations. Recruits were reared in a common garden for 5 weeks and then subjected to two thermal treatments (28  and 31 °C) for 2.5 weeks. The most significant effects were detected during the first 5 weeks, before thermal stress was applied: 27–30% of variance in growth and 94% of variance in recruit survival was attributable to parental effects. Genotyping of eight microsatellite loci indicated that the high early mortality of some of the recruit families was not due to higher inbreeding. Post treatment, parental effects diminished such that only 10–15% of variance in growth rate was explained, which most likely reflects the dissipation of maternal effects. However, offshore-origin recruits still grew significantly less under elevated temperature compared with inshore-origin recruits. These differences observed in naive juvenile corals suggest that population-level variation in fitness in response to different thermal environments has a genetic basis and could represent raw material for natural selection in times of climate change.