Finding new collective electronic states in materials is one of the fundamental goals of condensed matter physics. Atomic-scale superlattices formed from transition metal oxides are a particularly appealing hunting ground for new physics. In bulk form, transition metal oxides exhibit a remarkable range of magnetic, superconducting, and multiferroic phases that are of great scientific interest and are potentially capable of providing innovative energy, security, electronics and medical technology platforms. In superlattices new states may emerge at the interfaces where dissimilar materials meet. Here we illustrate the essential features that make transition metal oxide-based heterostructures an appealing discovery platform for emergent properties with a few selected examples, showing how charge redistributes, magnetism and orbital polarization arises and ferroelectric order emerges from heterostructures comprised of oxide components with nominally contradictory behavior with the aim providing insight into the creation and control of novel behavior at oxide interfaces by suitable mechanical, electrical or optical boundary conditions and excitations.