Theory predicts that positive Heterozygosity-Fitness Correlations (HFC) arise as a consequence of inbreeding, which is often assumed to have a strong impact in small, fragmented populations. Yet according to empirical data, HFC in such populations seem highly variable and impredictible. We here discuss two overlooked phenomena that may contribute to this variation. First, in a small population, each generation may consist of a few families. This generates random correlations between particular alleles and fitness (AFC, Allele-Fitness Correlations) and results in too liberal tests for HFC. Second, in some contexts, small populations receiving immigrants may be more impacted by outbreeding depression than by inbreeding depression, resulting in negative rather than positive HFC. We investigated these processes through a case study in tadpole cohorts of Pelodytes punctatus living in small ponds. We provide evidence for a strong family structure and significant AFC in this system, as well as an example of negative HFC. By simulations, we show that this negative HFC cannot be a spurious effect of family structure, and therefore reflects outbreeding depression in the studied population. Our example suggests that a detailed examination of AFC and HFC patterns can provide valuable insights into the internal genetic structure and sources of fitness variation in small populations.