What did science make possible for colonial rule? How was science in turn marked by the knowledge and practices of those under colonial rule? Here I approach these questions via the social history of Madras Observatory. Constructed in 1791 by the East India Company, the observatory was to provide local time to mariners and served as a clearinghouse for the company’s survey and revenue administration. The astronomical work of Madras’ Brahmin assistants relied upon their knowledge of jyotiśāstra [Sanskrit astronomy/astrology], and can be seen as a specialized form of the kind of South Indian scribal labor and knowledge that also staffed the company’s tax offices. If at Greenwich the division of labor meant observatory work bore resemblances to the factory and the accounts office, in Madras, astronomy and accounting drew on similar labor forms because they were part of the same enterprise. But the company did not just adapt preexisting forms of labor, it also attempted to produce its own at a school built near the observatory to train “half-caste” orphans as apprentice surveyors and assistant computers. The school, staffed by the Brahmins, drew upon knowledge and pedagogical practice associated with the tinnai, the schools in which upper-caste children learned to read, write, and calculate. For a time, the observatory’s social order was literally “half-caste.” The paper also considers how the relationship between caste, status, and instrument was reflected in the visual and material culture of the observatory, such as in Indian-language inscriptions on its central pillar. For company astronomers, the measurement of time meant reworking the relationships among the Indian past, the colonial present, and an imperial posterity. Science under colonial rule spanned multiple temporal and social registers because it was the result of negotiations between the demands of political economy and the knowledge and practices of colonized others.