Adaptation involves the successive substitution of beneficial mutations by selection, a process known as an adaptive walk. Gradualist models of adaptation, which assume that all mutations are small relative to the distance to a fitness optimum, predict that adaptive walks should be longer when the founding genotype is less well adapted. More recent work modelling adaptation as a sequence of moves in phenotype or genotype space predicts, by contrast, much shorter adaptive walks irrespective of the fitness of the founding genotype. Here we provide what is, to the best of our knowledge, the first direct test of these alternative models, measuring the length of adaptive walks in evolving lineages of fungus that differ initially in fitness. Contrary to the gradualist view, we show that the length of adaptive walks in the fungus Aspergillus nidulans is insensitive to starting fitness and involves just two mutations on average. This arises because poorly adapted populations tend to fix mutations of larger average effect than those of better-adapted populations. Our results suggest that the length of adaptive walks may be independent of the fitness of the founding genotype and, moreover, that poorly adapted populations can quickly adapt to novel environments.