Accounting for gene-environment (GxE) interactions in complex trait association studies can facilitate our understanding of genetic heterogeneity under different environmental exposures, improve the ability to discover susceptible genes that exhibit little marginal effect, provide insight into the biological mechanisms of complex diseases, help to identify high-risk subgroups in the population, and uncover hidden heritability. However, significant GxE interactions can be difficult to find. The sample sizes required for sufficient power to detect association are much larger than those needed for genetic main effects, and interactions are sensitive to misspecification of the main effects model. These issues are exacerbated when working with binary phenotypes and rare variants, which bear less information on association. In this work, we present a similarity-based regression method for evaluating GxE interactions for rare variants with binary traits. The proposed model aggregates the genetic and GxE information across markers using genetic similarity thus increasing the ability to detect GxE signals. The model has a random effects interpretation, which leads to robustness against main effect misspecifications when evaluating GxE interactions. We construct score tests to examine GxE interactions and a computationally efficient EM algorithm to estimate the nuisance variance components. Using simulations and data applications, we show that the proposed method is a flexible and powerful tool to study the GxE effect in common or rare variant studies with binary traits.