The aim of this study was to investigate the association between average 24-hour ambulatory heart rate and all-cause mortality, while adjusting for resting clinical heart rate, cardiorespiratory fitness, occupational and leisure time physical activity as well as classical risk factors. A group of 439 middle-aged male workers free of baseline coronary heart disease from the Belgian Physical Fitness Study was included in the analysis. Data were collected by questionnaires and clinical examinations from 1976 to 1978. All-cause mortality was collected from the national mortality registration with a mean follow-up period of 16.5 years, with a total of 48 events. After adjustment for all before mentioned confounders in a Cox proportional hazards regression analysis, a significant increased risk for all-cause mortality was found among the tertile of workers with highest average ambulatory heart rate compared to the tertile with lowest ambulatory heart rate (Hazard ratio = 3.21, 95% confidence interval: 1.22–8.44). No significant independent association was found between resting clinic heart rate and all-cause mortality. The study indicates that average 24-hour ambulatory heart rate is a strong predictor of all-cause mortality independent from resting clinic heart rate, cardiorespiratory fitness, occupational and leisure time physical activity and other classical risk factors among healthy middle-aged workers.