Cortical inhibitory neurons contact each other to form a network of inhibitory synaptic connections. Our knowledge of the connectivity pattern underlying this inhibitory network is, however, still incomplete. Here we discover a simple and complementary interaction scheme between three large molecularly distinct interneuron populations in mouse visual cortex: Parvalbumin expressing interneurons strongly inhibit one another but, surprisingly, provide little inhibition to other populations. In contrast, somatostatin expressing interneurons avoid inhibiting one another, yet strongly inhibit all other populations. Finally, vasoactive intestinal peptide expressing interneurons preferentially inhibit somatostatin interneurons. This scheme occurs in supra- and infra-granular layers, suggesting that inhibitory networks operate similarly at the input and output of visual cortex. Thus, as the specificity of connections between excitatory neurons forms the basis for the cortical canonical circuit, the scheme described here outlines a standard connectivity pattern among cortical inhibitory neurons.