Objectives: Non-restorative sleep (NRS) is related to qualitative aspects of sleep. The associations of NRS with exercise (EX; a planned and purposeful activity) and physical activity (PA; daily bodily movement) by gender and age has not yet been clarified. We investigated the associations of EX and/or PA with NRS by gender and age. Design: A cross-sectional study. Setting: The data on gender, age, presence of NRS, and engagement in EX and/or PA were obtained from database and questionnaire of specified medical checkups in FY 2013 in Japan. The analysis was conducted in 2017. Participants: The subjects comprised 90,122 residents (38,603 males and 51,519 females), aged 40 to 74 years, who had completed the specified medical checkups. Outcome measure: The presence of NRS was assessed using a question asking whether or not the subjects usually got enough sleep. NRS was considered to be present when the subjects answered “No”. Binary logistic regression analysis was used to assess the associations of presence or absence of EX and/or PA with NRS. The OR and 95% CI of NRS prevalence were calculated and compared between those engaged in both EX and PA and the others. Results: Except for 40s and 70s among males and 40s and 50s among females, the absence of EX or PA was associated with higher ORs of NRS than referent. ORs were more than twice compared to the referents in males in their 50s (OR 2.030 [95% CI 1.675, 2.459]) and 60s (OR 2.148 [95% CI 1.970, 2.343]) and females in their 60s (OR 2.142 [95% CI 1.994, 2.302]) when they engaged in neither EX nor PA. Conclusions: health care providers must take into account the similarities and differences in the associations of EX and/or PA with NRS by gender and age when they support people with NRS.