While colors are commonplace in everyday metaphors, relatively little is known about implicit color associations to linguistic or semantic concepts in a general population. In this study, we test color associations for ordered linguistic concepts (letters and days). The culture and language specificity of these effects was examined in a large group (457) of Dutch-speaking participants, 92 English-speaking participants, and 49 Hindi-speaking participants. Non-random distributions of color choices were revealed; consistencies were found across the three language groups in color preferences for both days and letters. Interestingly, while the Hindi-speaking participants were presented with letter stimuli matched on phonology, their pattern of letter-to-color preferences still showed similarities with Dutch- and English-speaking participants. Furthermore, we found that that the color preferences corresponded between participants indicating to have conscious color experiences with letters or days (putative synesthetes) and participants who do not (non-synesthetes). We also explored possible mechanisms underlying the color preferences. There were a few specific associations, including red for “A,” red for “Monday,” and white for “Sunday.” We also explored more general mechanisms, such as overall color preferences as shown by Simner et al. ( 2005). While certainly not all variation can be explained or predicted, the results show that regularities are present in color-to-letter or color-to-day preferences in both putative synesthetes and non-synesthetes across languages. Both letter-to-color and day-to-color preferences were influenced by multiple factors. The findings support a notion of abstract concepts (such as days and letters) that are not represented in isolation, but are connected to perceptual representational systems. Interestingly, at least some of these connections to color representations are shared across different language/cultural groups.