The large-scale conversion of natural mangroves to aquaculture reduces species richness and diversity. Large areas of abandoned aquaculture ponds in areas where mangroves formerly predominated in China and Southeast Asia represent important potential effective targets for mangrove restoration. Here, we empirically assessed the α-diversity (species richness) and β-diversity (variation in community composition) of mangrove, macrobenthos, fish, and waterbird in a tropical mangrove bay on Hainan Island, China. We compared sites subjected to different pond-to-mangrove restoration programs more than 20 years ago (passive restoration without planting, and active restoration with planting) to nearby reference site with natural mangrove forests and mudflats. To better understand how β-diversity responds to restoration, we also distinguished between β-diversity turnover and nestedness (richness difference). In general, α-diversity values for both fish and waterbird communities, and β-diversity values for the mangrove, macrobenthos, and waterbird communities were lower at the restoration sites than at the reference site, suggesting that the strong homogenizing effects of anthropogenic habitat alternation was still apparent after more than 20 years since aquaculture ceased. In addition, spatial turnover, not nestedness, dominated total β-diversity both across the whole study area and at individual sites, suggesting that multiple processes, such as environmental filtering, helped to shape multi-taxa community structures. Moreover, we found no evidence that planting in the abandoned ponds, in addition to standard hydrological restoration, supported greater species diversity of taxa like macrobenthos and waterbird than the naturally regenerated site after more than 20 years’ recovery. Our results underline the importance of avoiding the conversion of natural mangrove stands to aquaculture wherever possible, and the urgent need to design effective mangrove restoration techniques in tropical Asia.