The study of modularity is paramount for understanding trends of phenotypic evolution, and for determining the extent to which covariation patterns are conserved across taxa and levels of biological organization. However, biologists currently lack quantitative methods for statistically comparing the strength of modular signal across datasets, and a robust approach for evaluating alternative modular hypotheses for the same dataset. As a solution to these challenges, we propose an effect size measure (Z_CR) derived from the covariance ratio, and develop hypothesis-testing procedures for their comparison. Computer simulations demonstrate that Z_CR displays appropriate statistical properties and low levels of misspecification, implying that it correctly identifies modular signal, when present. By contrast, alternative methods based on likelihood (EMMLi) and goodness of fit (MINT) suffer from high false positive rates and high model misspecification rates. An empirical example in sigmodontine rodent mandibles is provided to illustrate the utility of Z_CR for comparing modular hypotheses. Overall, we find that covariance ratio effect sizes are useful for comparing patterns of modular signal across datasets or for evaluating alternative modular hypotheses for the same dataset. Finally, the statistical philosophy for pairwise model comparisons using effect sizes should accommodate any future analytical developments for characterizing modular signal.