Collective emotion has been traditionally evaluated by questionnaire survey on a limited number of people. Recently, big data of written texts on the Internet has been available for analyzing collective emotion for very large scales. Although short-term reflection between collective emotion and real social phenomena has been widely studied, long-term dynamics of collective emotion has not been studied so far due to the lack of long persistent data sets. In this study, we extracted collective emotion over a 10-year period from 3.6 billion Japanese blog articles. Firstly, we find that collective emotion shows clear periodic cycles, i.e., weekly and seasonal behaviors, accompanied with pulses caused by natural disasters. For example, April is represented by high Tension, probably due to starting school in Japan. We also identified long-term memory in the collective emotion that is characterized by the power-law decay of the autocorrelation function over several months.