To examine the relationship between social contextual factors and child and adolescent labor. Population-based cohort study carried out with 2,512 families living in 23 subareas of a large urban city in Brazil from 2000 to 2002. A random one-stage cluster sampling was used to select families. Data were obtained through individual household interviews using questionnaires. The annual cumulative incidence of child and adolescent labor was estimated for each district. New child and adolescent labor cases were those who had their first job over the two-year follow-up. The annual cumulative incidence of child and adolescent labor was the response variable and predictors were contextual factors such as lack of social support, social deprivation, unstructured family, perceived violence, poor school quality, poor environment conditions, and poor public services. Pearson's correlation and multiple linear regression were used to assess the associations. There were selected 943 families corresponding to 1,326 non-working children and adolescents aged 8 to 17 years. Lack of social support, social deprivation, perceived violence were all positively and individually associated with the annual cumulative incidence of child and adolescent labor. In the multiple linear regression model, however, only lack of social support and perceived violence in the neighborhood were positively associated to child and adolescent labor. No effect was found for poor school quality, poor environment conditions, poor public services or unstructured family. Poverty reduction programs can reduce the contextual factors associated with child and adolescent labor. Violence reduction programs and strengthening social support at the community level may contribute to reduce CAL.