The uncontrolled cellular growth that characterizes tumor formation requires a constant delivery of nutrients. Since the 1970s, researchers have wondered if the supply of nutrients from the diet could impact tumor development. Numerous studies have assessed the impact of dietary components, specifically sugar and fat, to increased cancer risk. For the most part, data from these trials have been inconclusive; however, this does not indicate that dietary factors do not contribute to cancer progression. Rather, the dietary contribution may be dependent on tumor, patient, and context, making it difficult to detect in the setting of large trials. In this review, we combine data from prospective cohort trials with mechanistic studies in mice to argue that fat and sugar can play a role in tumorigenesis and disease progression. We find that certain tumors may respond directly to dietary sugar (colorectal and endometrial cancers) and fat (prostate cancer) or indirectly to the obese state (breast cancer).