Although different gender associations between self-rated health (SRH) and mortality have been reported, the results of the respective studies have been inconsistent and little is known about the cause-specific relation of mortality with SRH by gender. Therefore, to evaluate the gender differences in all-cause or specific causes of mortality by SRH, this retrospective cohort study was conducted using the data of 19,770 Korean adults aged 50 years and over who underwent health screening at Seoul National University Hospital between March 1995 and December 2008. SRH was surveyed using a simple questionnaire, and the all-cause mortality and cause-specific mortality were followed up from baseline screening until December 31, 2016. Results showed that the relationship between SRH and all-cause mortality differed by gender, and the differences also varied depending on the cause of death. In men, the adjusted hazard ratio (aHR) of all-cause mortality was higher in the poor SRH group than the very good SRH groups even after adjustment for socio-demographic, clinical, and behavioral risk factors (aHR:1.97, 95% CI 1.51–2.56), and these results were similar to those for cancer, cardiovascular, and respiratory disease mortalities (aHR:1.52, 95% CI 0.93–2.50; aHR: 2.11, 95% CI 1.19–3.74; aHR:10.30, 95% CI 2.39–44.44, respectively). However, in women, the association between SRH and all-cause mortality was insignificant, and inverse relationships were found for cardiovascular and respiratory disease mortalities in the poor and very good SRH groups. Cancer mortality had a positive relation with SRH (aHR: 1.14, 95% CI 0.75–1.72; aHR: 2.58, 95% CI 1.03–6.48; aHR: 0.49, 95% CI 0.24–0.98; aHR: 0.15, 95% CI 0.04–0.57: all-cause, cancer, cardiovascular, and respiratory disease mortalities, respectively). Clinicians need to take these gender differences by SRH into account when evaluating the health status of over-middle aged adults.